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Abstract 

The  purpose  of  this  research  is  to  analyze  competencies  needed  for  successful 
program  management  of  Research  and  Development  (R&D)  programs — also  known  as 
Science  and  Technology  (S&T)  programs  within  the  Department  of  Defense  (DoD). 
Current  competency  models  are  in  development  for  the  DoD  Acquisition,  Technology, 
and  Logistics  (AT&L)  functional  areas.  Using  an  interim  competency  model  developed 
for  the  Program  Manager  career  field  comprised  of  36  technical  competencies  and  27 
professional  competencies,  the  research  compares  competencies’  criticality  scores  rated 
by  traditional  acquisition  program  managers  (PMs)  with  those  rated  by  S&T  program 
managers.  In  42  out  of  63  instances  (67%)  the  criticality  scores  had  statistically 
significant  differences.  Only  four  of  those  42  competencies  were  rated  “more  critical”  by 
S&T  PMs;  the  other  38  of  the  42  (90%),  were  rated  “less  critical”  with  statistically 
significant  lower  scores  than  those  of  their  acquisition  PM  counterparts.  The  analysis  of 
the  open-ended  interview  questions  suggests  that  although  the  AT&L  PM  competency 
model  may  seem  to  have  face  validity,  the  descriptions  currently  defining  the  PM 
competencies  may  not  have  adequate  content  validity  for  an  S&T  program  management 
competency  model,  thus  warranting  further  resources  towards  defining  an  independent 
competency  model  for  S&T  PM  workforce  management  initiatives. 
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Glossary  of  Terms 


1 .  Acquisition,  Technology  and  Logistics  (AT&L)  Workforce:  all  individuals  who 
occupy  AT&L  positions,  to  more  accurately  reflect  the  breadth  of  the  types  of 
functions  and  duties  performed  by  employees 

Desk  Guide  for  AT&L  Workforce  Career  Management 

2.  Competency:  an  observable  and  measurable  pattern  of  knowledge,  skills,  abilities, 
behaviors,  and  other  personal  characteristics  that  an  individual  needs  to  perform 
work  roles  or  occupational  functions  successfully 

-  Office  of  Personnel  Management 

3.  Competency  Model:  a  framework  that  describes  the  full  range  of  competencies 
required  to  be  successful  in  a  particular  occupation 

-  Office  of  Personnel  Management 

4.  Program:  a  group  of  related  projects  managed  in  a  consolidated  way 

-  Program  Management  Institute 

5.  Program  Management:  centralized  coordinated  management  of  a  group  of 
projects  to  achieve  a  program’s  strategic  goals 

-  Program  Management  Institute 
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AN  ANALYSIS  OF  COMPETENCIES  FOR  MANAGING 
SCIENCE  AND  TECHNOLOGY  PROGRAMS 

I.  Introduction 

Staying  cognizant  of  and  being  responsive  to  the  multitude  of  environmental 
changes  which  can  shape  a  profession’s  proficiency  standards  are  constant  challenges, 
especially  in  today’s  fast-paced  and  technologically  evolving  workplace.  The 
Department  of  Defense  (DoD)  Acquisition,  Technology,  and  Logistics  (AT&L) 
community  and  in  particular  its  program  managers  (PMs)  have  received  much 
congressional  and  media  attention  regarding  proficiency  challenges  over  the  last  two 
decades.  Tremendous  pressure  from  public  scrutiny  has  demanded  that  PMs  perfonn 
with  competency,  adaptability,  and  accountability,  regardless  of  the  myriad  positions  they 
might  assume  across  AT&L  organizations  or  program  lifecycle  phases.  Thus,  many 
studies  and  best-practices  about  the  profession  of  program  management — or  project 
management,  as  it  is  commonly  termed  in  private  industry — have  filled  professional 
journals  in  recent  years.  Nevertheless,  current  literature  suggests  the  AT&L  workforce 
still  faces  current  and  future  challenges  in  acquiring  and  maintaining  the  right  mixture  of 
expertise  required  for  proficient  program  management  in  today’s  quickly  changing  work 
environment.  Therefore,  this  research  will  attempt  to  analyze  technical  and  professional 
competencies  identified  for  superior  PM  perfonnance  within  DoD,  specifically 
comparing  those  identified  for  traditional  Acquisition  program  management  with  those 
needed  for  Science  and  Technology  (S&T)  program  management. 
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The  purpose  of  this  chapter  is  to  outline  the  efforts  of  this  research  by  introducing 
the  background  of  the  Department  of  Defense  (DoD)  Acquisition,  Technology,  & 
Logistics  (AT&L)  workforce  environment  that  is  driving  competency  modeling 
initiatives,  the  scope  and  methodology  for  this  research,  and  the  expected  benefits  and 
limitations  of  the  research  results. 

DoD  AT&L  Background 

The  last  two  decades  have  witnessed  many  challenges  to  the  DoD  AT&L 
community.  In  the  aftennath  of  acquisition  scandals  based  on  individual  integrity 
breaches,  program  cost/schedule  overruns,  and  contract  award  disputes,  AT&L  leadership 
faces  tremendous  scrutiny  for  holding  its  workforce  accountable — particularly  those 
given  the  enormous  responsibility  of  managing  its  costly  acquisition  programs.  At  the 
same  time,  DoD  faces  a  looming  talent  loss  with  the  projected  retirement  of  its  aging 
workforce  and  a  nation-wide  competition  for  dwindling  expertise  and  experience 
(Department  of  Defense  Acquisition,  Technology  and  Logistics,  2007a:7).  Thus,  multiple 
agencies  have  levied  directives  for  new  management  initiatives  to  produce  a  high- 
performing,  agile  workforce  to  bring  the  DoD  AT&L  mission  successfully  through  the 
myriad  challenges  ahead  (Hausmann  and  Tregar,  2006:  4): 

•  2002  President’s  Management  Agenda  focusing  on  five  areas  of 
management  weakness  across  the  government  for  improvement, 
specifically  identifying  human  perfonnance  management  systems 

•  2002  and  2004  Government  Accountability  Office  (GAO)  reports 
emphasizing  the  importance  of  competencies  for  improving  effectiveness 
and  addressing  employee  shortages  due  to  downsizing 
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•  2005  Office  of  Management  and  Budget  (OMB)  policy  letter 
recommending  changing  acquisition  certification  from  a  course-based  to  a 
competency-based  strategy 

•  2006  Quadrennial  Defense  Review  (QDR)  recommending  a  performance- 
based  approach  to  measuring  human  capital 

•  2007  DoD  AT&L  Human  Capital  Strategic  Plan  describing  the 
improvement  of  competency  management. 

Hence,  AT&L  and  its  component  agencies  are  investing  research  and  development  efforts 
towards  a  competency  model  by  which  organizations  can  identify,  define,  and  prioritize 
the  desired  competencies  into  their  various  workforce  management  programs. 

Competency  Modeling 

The  concept  of  competency  modeling  has  evolved  over  the  last  three  decades. 
Literature  attributes  the  genesis  of  the  concept  to  personality  and  social  psychologist 
David  McClelland,  who  in  1973  suggested  that  competencies — rather  than  intelligence — 
were  more  related  to  job  performance  outcomes,  thus  spurring  a  new  era  in  re-defining 
workforce  management  (Shippmann  and  others,  2000:7 1 1).  Twenty-five  years  later,  a 
diverse  group  of  eight  researchers  attempted  to  trace  the  evolution  of,  and  current 
standard  for,  competency  modeling  through  a  two-year  task  force  commissioned  by  the 
Society  for  Industrial  and  Organizational  Psychology  (Shippmann  and  others,  2000:704). 
This  Job  Analysis  and  Competency  Modeling  Task  Force  (JACMTF)  conducted  an 
extensive  literature  search  and  interviewed  subject  matter  experts  (SMEs)  regarding  the 
development  and  use  of  competency  models,  producing  a  comprehensive  overview  titled 
“The  Practice  of  Competency  Modeling”  (Shippmann  and  others,  2000:704).  Like  other 
literature,  the  task  force  attributes  the  boon  in  competency  modeling  within  industry  and 
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business  to  authors  C.K.  Prahalad  and  Gary  Hamel  and  their  1990  Harvard  Business 
Review  article  titled,  “The  Core  Competence  of  the  Corporation,”  which  became  the 
journal’s  most  reprinted  paper  in  history  to  date  (Shippmann  and  others,  2000:7 12). 

The  new  emphasis  on  individual-level  competencies  as  the  foundation  for  a 
business’s  core  competence  also  prompted  government  organizations  to  invest  resources 
towards  competency-focused  initiatives.  The  United  States  Office  of  Personnel 
Management  (OPM)  began  efforts  in  1990  which  have  continued  to  evolve  (Rodriguez 
and  others,  2002:3 10).  Faced  with  tremendous  changes  in  the  pace,  paradigms,  and 
practices  of  the  1990s,  organizations  recognized  “the  value  of  a  workforce  that  is  not  only 
highly  skilled  and  technically  adept,  but  more  importantly,  a  workforce  that  can  learn 
quickly,  adapt  to  change,  communicate  effectively,  and  foster  interpersonal  relationships” 
(Rodriguez  and  others,  2002:310).  OPM  also  “envisioned  a  unifonn,  competency-based 
common  language  that  would  enable  federal  agencies  to  describe  jobs  in  the  same  way, 
eliminating  inconsistencies  across  agencies  and  HR  functions”  and  thus  “promote  a 
common  understanding  of  the  critical  elements  of  each  job  among  HR  personnel, 
management,  and  employees”  (Rodriguez  and  others,  2002:31 1). 

These  aims  have  propelled  the  federal  government,  and  in  particular  the  Federal 
Acquisition  Institute  (FAI),  to  direct  studies,  validation,  and  execution  of  current 
competency  models  for  its  workforce.  Aside  from  the  regular  reviews  of  standards  and 
compliance  under  the  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  the 
FAI  “led  an  inter-agency  working  group  to  develop  common,  essential  competencies  for 
the  program  and  project  management  community”  (Office  of  Management  and  Budget, 
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2007:1).  Most  recently,  the  Office  of  the  Undersecretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  (OSD(AT&L))  issued  a  memorandum  to  its  AT&L  Program 
Management  (PM)  workforce  announcing  its  joint  competency  initiative  “to  establish  a 
standard  competency  model  for  each  career  field  in  the  DoD  AT&L  workforce”  in  order 
to  “map  the  array  of  competencies  and  perfonnance  criteria  required  to  be  successful  in 
the  acquisition  career  field”  (Ahern  and  Anderson,  2008:  1).  AT&L  Director  of  Portfolio 
Systems  Acquisition,  David  G.  Ahern,  and  Defense  Acquisition  University  (DAU) 
President,  Frank  J.  Anderson  (2008:1),  expect  the  initiative  to  allow  for  the  assessment 
and  refinement  of  “the  requisite  competencies  within  the  current  DoD  Program 
Management  workforce”  as  well  as  the  development  of  “appropriate  strategies  to  shape 
the  skill  sets  and  capabilities”  of  PMs  needed  in  the  future. 

Research  Question  &  Scope 

The  overarching  question  for  this  research  study  is  whether  program  management 
competencies  differ  across  the  AT&L  spectrum.  Using  the  current  AT&L  program 
management  competency  model,  one  investigative  question  is  whether  a  one-size-fits-all 
competency  model  for  program  management  is  sufficient,  or  whether  some  competencies 
are  more  or  less  critical  for  different  program  manager  positions — specifically  within 
Research  and  Development  (R&D)  programs.  If  so,  the  final  investigative  question  tests 
whether  those  differences  are  statistically  significant  enough  to  recommend  additional 
research  and  resources  towards  a  specialized  PM  competency  model  to  guide  current  and 
future  workforce  management  initiatives  for  R&D  program  managers. 
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For  purposes  of  this  research  scope,  only  competencies  identified  predominantly 
by  PMs  managing  traditional  acquisition  programs  and  those  identified  for  PMs 
managing  R&D — or  Science  and  Technology  (S&T) — programs  are  analyzed.  Because 
AT&L  uses  the  tenn  ‘Science  and  Technology  Manager’  (STM)  to  label  this  functional 
category,  ‘S&T’  will  be  the  term  used  throughout  this  research  and  assumed  synonymous 
with  R&D.  The  representative  sample  group  for  data  regarding  STM  competencies 
includes  personnel  from  the  Air  Force  Research  Laboratory  (AFRL)  at  Wright  Patterson 
Air  Force  Base,  OH.  AFRL  is  recognized  as  the  leading  organization  for  managing  S&T 
programs  within  the  Air  Force  and  in  collaboration  with  other  S&T  agencies  across  DoD, 
industry,  and  academia. 

Methodology 

The  methodology  for  this  research  follows  an  exploratory  approach  combining 
elements  of  qualitative  and  quantitative  design.  It  begins  with  a  literature  review  to 
explore  existing  research  about  the  AT&L  environment,  competency  modeling,  and 
program  management  career  field  issues.  Based  on  literature  reviews  and 
communications  with  DoD  program  management  career  field  stakeholders,  the  2006 
AT&L  Program  Management  Career  Field:  Interim  Competency  Model  is  determined  to 
be  the  most  relevant  study  upon  which  to  model  this  research.  To  achieve  the  most  valid 
results,  the  data  collection  process  attempts  to  mirror  a  subset  of  the  methodology 
outlined  in  the  AT&L  study  as  closely  as  feasible  for  the  portions  applicable  to  this 
research  scope.  The  AT&L  competency  model  incorporates  methodologies  consistent 
with  competency  modeling  practices  documented  in  current  literature:  “determine  what 
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leads  to  superior  performance  and  to  identify  top  performers  to  find  out  what  they  do” 
(Lucia  and  Lepsinger,  1999:18). 

Data  Collection. 

Like  the  AT&L  study,  the  first  of  a  three-step  data  collection  and  analysis  process 
relies  on  an  “Expert  Panel”  consisting  of  highly  knowledgeable  functional  leaders  to 
establish  a  baseline  set  of  competencies  and  to  select  the  superior-performing  subject 
matter  experts  (SMEs)  to  evaluate  those  competencies.  The  second  step  involves 
interviewing  SMEs  through  semi-structured  open-ended  questions  and  structured  closed- 
ended  questions.  The  data  collection  for  this  research  relies  on  SMEs  responding  to  three 
Likert  scale  questions  regarding  the  importance,  frequency,  and  experience  level  first 
used  for  36  technical  competencies  and  27  professional  competencies  as  previously 
identified  from  the  AT&L  study.  It  also  includes  three  open-ended  questions  of  interest 
to  this  research.  Although  the  AT&L  study’s  methodology  prescribes  a  final  validation 
survey,  this  third  step  will  remain  a  recommendation  for  future  work  extending  beyond 
the  scope  of  this  research  study. 

Data  Analysis. 

Once  data  are  collected  from  the  S&T  program  managers,  they  are  computed  into 
results  to  compare  with  those  from  the  AT&L  study.  Descriptive  statistics  are  analyzed 
within  and  between  the  two  study  samples,  then  inferential  statistics  are  used  to  analyze 
the  comparisons  through  hypothesis  testing  and  t-tests  relevant  to  the  research 
investigative  questions.  These  assessments,  combined  with  a  qualitative  assessment  of 


7 


the  interviewees’  responses  to  open-ended  questions,  become  the  basis  for  conclusions 
and  recommendations  for  future  research. 

Constraints  &  Limitations. 

The  exploratory  nature  of  this  study,  involving  qualitative  and  quantitative 
methods,  incurs  some  natural  and  research-induced  constraints  that  limit  generalizing  to 
the  population  at  large.  However,  the  specific  construct  validity  issues  addressed  in 
Chapters  3  and  4  actually  elucidate  opportunities  and  recommendations  for  improving  the 
development  of  competency  modeling  and  its  application  towards  shaping  workforce 
management  initiatives  within  Air  Force  Science  and  Technology  organizations. 

Expected  Benefits 

The  expected  benefit  of  this  exploratory  research  is  to  contribute  to  the 
development  of  AT&L  competency  models  for  program  management,  specifically  in 
support  of  superior  S&T  program  management  performance.  The  results  are  broadly 
relevant  to  workforce  management  issues  such  as  selection,  training  and  development, 
performance  evaluation,  and  succession  planning.  They  are  specifically  relevant  to  the 
Individual  Development  Plans  (IDPs)  of  current  and  future  S&T  program  managers. 
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II.  Literature  Review 


The  purpose  of  this  chapter  is  to  present  literature  addressing  the  background  of 
the  Department  of  Defense  (DoD)  Acquisition,  Technology,  &  Logistics  (AT&L) 
workforce  environment  and  current  issues,  along  with  the  evolution  and  current  practices 
of  competency  modeling. 

DoD  (AT&L)  Background 

The  DoD  budget  represents  the  largest  portion  of  discretionary  spending  by  the 
United  States  government.  Consequently,  taxpayers  duly  expect  their  government 
officials  to  hold  accountable  those  responsible  for  such  critical  expenditures.  The 
responsibilities  placed  on  the  DoD’s  cadre  of  military  and  civilian  AT&L  program 
managers  are  “enormous,”  to  include:  managing  complex  and  evolving  weapon  system 
configurations;  coordinating  across  a  “broad  array  of  military  service  and  DoD  officials, 
outside  suppliers,  internal  and  external  oversight  entities,  as  well  as  technical,  business, 
contracting,  and  management  expertise;”  and  incorporating  state-of-the-art  and  often 
untested  technologies  while  facing  global  logistics  and  environmental  challenges 
(Government  Accountability  Office,  2005:3).  However,  scandals  since  the  1980s  have 
cast  doubt  on  the  integrity  and/or  competency  of  DoD  AT&L  personnel,  prompting 
Congress  and  the  media  to  require  regular  measures  of  public  accountability  for  program 
management  performance  and  results.  In  response,  the  DoD  has  launched  several 
“Acquisition  Reform”  and  “Transformation”  initiatives  since  the  early  1990s  targeting  its 
AT&L  workforce. 
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AT&L  Workforce  Environment 

One  initiative  stemming  from  the  1989  Defense  Management  Report  was  the 
1990  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  which  was  initially 
enacted  by  Public  Law  101-510  under  Title  10  of  the  U.S.  Code  and  intended  to 
“improve  the  effectiveness  of  the  personnel  who  manage  and  implement  defense 
acquisition  programs”  (Defense  Acquisition  University,  2006).  Furthermore,  it  called  for 
“establishing  an  Acquisition  Corps  and  professionalizing  the  acquisition  workforce 
through  education,  training,  and  work  experience”  as  part  of  the  1991  Defense 
Authorization  Act.  Periodic  changes  have  been  made  to  the  DAWIA  initiative  since  its 
inception,  with  updates  and  status  reports  publicized  in  literature  and  presented  to 
Congress. 

About  five  years  after  DAWIA ’s  initiation,  DoD  officials  reported  that  since  its 
provisions  had  gone  into  effect  in  October  1993,  each  military  department  had 
established  an  integrated  military  and  civilian  acquisition  corps  with  regulated 
qualifications  and  performance  standards  (Department  of  Defense,  1995).  Although  they 
met  initial  compliance  regulations  and  improved  workforce  development  programs,  each 
service  had  developed  its  separate  policies  and  implementation  strategies — such  as 
position  selections  and  mobility  requirements — despite  DAWIA  goals  of  uniformity 
across  the  acquisition  career  field  (Garcia,  Keyner,  Robillard,  and  Van  Mullecom, 
1997:295).  Thus,  continued  disparity  and  recurring  acquisition  scandals  prompted 
additional  workforce  studies  and  reports. 
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The  2005  Government  Accountability  Office  (GAO)  report  titled  Best  Practices: 
Better  Support  of  Weapon  System  Program  Managers  Needed  to  Improve  Outcomes, 
addressed  specific  flaws  impairing  workforce  success,  suggesting  that  senior  leader 
support  and  disciplined  knowledge-based  processes  were  two  critical  enablers  still 
lacking  in  DoD  program  management  (Government  Accountability  Office,  2005:4). 
Continued  concern  over  the  challenges  of  ensuring  a  robust  acquisition  workforce 
penneated  all  federal  organizations.  In  its  2005  Policy  Letter  05-01  to  the  Heads  of 
Civilian  Executive  Departments  and  Agencies,  the  Office  of  Management  and  Budget 
(OMB)  acknowledged  that  “the  quality  and  effectiveness  of  the  federal  acquisition 
process  depend  on  the  development  of  a  capable  and  competent  workforce”  (Office  of 
Management  and  Budget,  2005:  1).  Furthermore,  it  reemphasized  the  Services 
Acquisition  Reform  Act  of 2003  expansion  of  the  definition  of  acquisition  workforce  to 
include  all  individuals  who  perform  acquisition-related  functions:  “traditional  contracting 
functions,  requirements  definition,  measurement  of  contract  performance,  and  technical 
and  management  direction”  for  the  sole  purpose  of  ensuring  such  individuals  would  be 
“trained  and  developed  using  common  standards”  (Office  of  Management  and  Budget, 
2005:  1).  Lastly,  OMB  tasked  the  DoD  to  establish  a  framework  of  core  competencies 
and  develop  “specialized  competencies  for  particular  areas  of  focus”  (Office  of 
Management  and  Budget,  2005:  4). 

Thus,  the  DoD  began  its  2005-2006  Defense  Acquisition  Performance 
Assessment  (DAP A)  Project.  Senior  acquisition  leaders  and  external  consultants  planned 
and  executed  a  meticulous  methodology  to  identify  performance  improvements: 
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We  reviewed  over  1,500  documents  to  establish  a  baseline  of  previous 
acquisition  reform  recommendations,  held  open  meetings  and  operated  a 
public  web  site  to  obtain  public  input,  heard  from  107  experts,  received 
over  170  hours  of  briefings,  conducted  a  detailed  survey  and  interviews  of 
over  130  government  and  industry  acquisition  professionals,  and 
subsequently  developed  1,069  observations.  From  these  specific 
assessments,  we  identified  necessary  performance  improvements  and 
defined  implementation  criteria  for  each  area  of  improvement. 

(DAPA  Panel,  2005:2) 

Acknowledging  the  current  situation  as  “characterized  by  massively  accelerated  cost 
growth  in  major  defense  programs,  lack  of  confidence  by  senior  leaders,  and  no 
appreciable  improvement  in  the  defense  acquisition  system”  in  two  decades,  Acting 
Deputy  Secretary  of  Defense  Gordon  England  set  the  following  DAPA  Project  goals: 
“improve  the  DoD’s  acquisition  system  to  provide  capabilities  to  win  the  global  war  on 
terror;  meet  other  challenges  to  national  security;  and  regain  senior  leadership 
confidence”  (Department  of  Defense,  2006). 

Contributing  towards  these  goals,  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (AT&L)  also  commissioned  the  Defense  Science 
Board  (DSB)  to  evaluate  DoD  AT&L  business  processes  and  human  resources  capital 
(Defense  Science  Board,  2006:5).  The  DSB  panel  reported  “the  acquisition  system  is  the 
weak  link  in  the  transformation  chain”  (Defense  Science  Board,  2006:13)  with  ineffectual 
acquisition  reform  efforts  along  with  “inefficient  and  inflexible”  career  rules  based  on  a 
“one-size-fits-all  model  of  arbitrary  career  profiles”  (Defense  Science  Board,  2006:36). 
The  report  also  suggested  the  system  “invests  inefficiently  in  education  and  training,  and 
needs  to  be  better  synchronized  with  career  paths”  (Defense  Science  Board,  2006:36). 
These  workforce  development  deficiencies  were  again  highlighted  by  Government 
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Accountability  Office  (GAO)  testimony  in  2007  before  the  Committee  on  Homeland 
Security  and  Governmental  Affairs,  U.S.  Senate,  entitled  Federal  Acquisitions  and 
Contracting:  Systemic  Challenges  Need  Attention  (Government  Accountability  Office, 
2007:1).  The  testimony  stated  the  government  must  develop  an  accountable  and  capable 
AT&L  workforce  responsible  for  strategic  planning  and  management  of  programs  and 
contracts — especially  as  the  “workload  and  complexity  of  responsibilities  have  been 
increasing  without  adequate  attention  to  the  workforce’s  size,  skills  and  knowledge,  and 
succession  planning”  (Government  Accountability  Office,  2007: 1). 

To  better  address  these  issues,  the  previously  four  independent  acquisition 
services  consolidated  into  a  single  integrated  Defense  Acquisition  Corps  and  established 
the  AT&L  Workforce  Senior  Steering  Board  (SSB)  to  oversee  an  AT&L  Workforce 
Management  Group  (Department  of  Defense,  2006b:  4).  The  DoD  also  expanded  its 
previously  limited  notion  of  “acquisition  workforce”  under  the  term  “AT&L  Workforce” 
to  include  all  individuals  who  occupy  AT&L  positions  to  “more  accurately  reflect  the 
breadth  of  the  types  of  functions  and  duties  performed  by  employees”  (Department  of 
Defense,  2006b:  1).  This  initiative  further  specified  particular  areas  of  focus  according  to 
the  following  AT&L  position  category  functions  outlined  in  Table  1. 
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Table  1:  AT&L  Position  Categories 


AT&L  Position  Category 

Code 

Auditing 

u 

Business,  Cost  Estimating,  and  Financial  Management 

K 

Contracting 

C 

Education,  Training,  and  Career  Development 

X 

Facilities  Engineering 

F 

Industrial/Contract  Property  Management 

D 

Information  Technology 

R 

Life  Cycle  Logistics 

L 

Production,  Quality  and  Manufacturing 

H 

Program  Management 

A 

Program  Management  Oversight 

V 

Purchasing 

E 

Systems  Planning,  Research,  Development,  Engineering  -- 

Science  and  Technology  Manager 

1 

Systems  Planning,  Research,  Development,  Engineering  — 
Systems  Engineering 

S 

Test  and  Evaluation 

T 

Within  each  of  these  workforce  functions,  AT&L  initiated  efforts  towards  implementing 
the  2005  Office  of  Management  and  Budget  (OMB)  recommendation  to  change  the 
certification  of  the  acquisition  community  “from  a  course-based  to  a  competency-based 
strategy”  (Hausmann  and  Tregar,  2006:  5).  According  to  the  Defense  Acquisition 
Transformation  Report  to  Congress,  pursuant  to  section  804  of  the  John  Warner  National 
Defense  Authorization  Act  for  Fiscal  Year  2007,  “a  sense  of  urgency  has  been 
established  by  the  Department  to  streamline  and  simplify  the  Acquisition  System  with 
aggressive  initiatives  to  provide  lasting  solutions  for  predictable  performance”  (Office  of 
the  Secretary  of  Defense,  2007:3). 
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Additional  workforce  perfonnance  initiatives  are  outlined  in  the  2007  AT&L 
Human  Capital  Strategic  Plan  v.  3.0,  claiming  “people”  as  the  department’s  most 
important  asset,  with  commitment  to  maintaining  a  “high  performing,  agile,  and  ethical 
workforce”  as  the  number  one  priority  (Krieg,  2007).  The  3.0  version  added  a  sixth  goal 
entitled  “Recruit,  develop,  and  retain  a  mission-ready  workforce  through  comprehensive 
talent  management”  as  outlined  in  Table  2  (Office  of  the  Secretary  of  Defense,  2007:1 1). 


Table  2:  AT&L  Human  Capital  Strategic  Plan  v3.0  Goals 


AT&L  Human  Capital  Strategic  Plan  v3.0  Goals 

1 

Align  and  fully  integrate  with  overarching  DoD  human  capital 
initiatives 

2 

Maintain  a  decentralized  execution  strategy  that  recognizes 
the  Component  leaders'  lead  role  and  responsibility  for  force 
planning  and  workforce  management 

3 

Establish  a  comprehensive,  data-driven  workforce  analysis 
and  decision-making  capability 

4 

Provide  learning  assets  at  the  point  of  need  to  support 
mission-responsive  human  capital  development 

5 

Execute  DoD  AT&L  workforce  communications  plan  that  is 
owned  by  all  AT&L  senior  leaders  (One  Team,  One  vision,  A 
Common  Message,  and  Integrated  Strategies) 

6 

Recruit,  develop,  and  retain  a  mission  ready  workforce 
through  comprehensive  talent  management 

Regarding  the  fourth  goal,  the  Undersecretary  of  Defense  for  AT&L  states  that  the  “rapid 
pace  of  change  with  learning  concepts  and  technologies  has  enabled  us  to  help  our 
workforce  learn  and  be  successful  on  the  job  by  delivering  the  right  knowledge  and  skills 
at  the  point  of  need”  (Office  of  the  Secretary  of  Defense,  2007:9).  Thus,  in  alignment 
with  the  DAPA  Project,  GAO  reports,  DSB  recommendations,  and  senior  leader 
direction,  OSD(AT&L)  workforce  committees  are  continuing  to  pursue  strategies  for 


15 


ensuring  AT&L  professionals  are  better  organized,  trained,  and  equipped  to  achieve 
optimum  perfonnance  potential  specific  to  their  AT&L  functional  areas. 

Current  Workforce  Issues. 

Identifying  and  prioritizing  the  requirements  for  each  type  of  program/project 
manager  position  across  the  AT&L  spectrum  becomes  essential  to  effectively  hiring, 
developing,  and  retaining  competent  employees.  Although  each  DoD  component 
structures  its  acquisition  organizations  differently,  their  program  managers  fulfill  similar 
roles  and  responsibilities  across  the  defense  acquisition  management  framework  spanning 
multiple  levels  of  product  maturity:  the  pre-acquisition  phase  consists  of  concept 
refinement  and  technology  development;  the  systems  acquisition  phase  consists  of  system 
development  and  demonstration  plus  production  and  deployment;  and  the  sustainment 
phase  consists  of  operations  and  support  (Department  of  Defense,  2003:2).  Despite 
differences  in  mission  requirements  and  challenges  across  these  phases,  AT&L  program 
managers  have  traditionally  been  developed  under  the  same  Acquisition  Professional 
Development  Program  (APDP)  certification  model.  For  new  program  managers  with 
lower  levels  of  responsibility  and  under  the  supervision  of  experienced  PMs,  sufficient 
time  and  opportunity  exist  within  the  traditional  APDP  model  for  developing  the  skills, 
knowledge,  and  abilities  (SKAs)  for  higher  levels  of  competency.  However,  with  50%  of 
its  AT&L  workforce  becoming  retirement  eligible  over  the  next  five  years  (Department 
of  Defense  Acquisition,  Technology,  and  Logistics,  2007a:  10),  the  DoD  needs  strategies 
for  mitigating  the  impacts  of  losing  this  specialized  expertise  and  for  maintaining  a 
proficient  and  adaptable  “bench  strength”  (35). 
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In  a  proactive  attempt  to  mitigate  this  projected  experience  loss,  in  Fiscal  Years 
2002-2006  the  DoD  targeted  7,140  new  hires  in  the  Systems  Planning,  Research, 
Development  and  Engineering  (SPRDE)  functional  area  (comprised  of  Science  and 
Technology  Managers  and  Systems  Engineering  careers  fields)  and  1,338  new  hires  in 
Program  Management  (PM),  representing  a  combined  45%  of  all  new  hires  as  outlined  in 
Table  3  (Office  of  the  Secretary  of  Defense,  2007:12). 


Table  3:  DoD  Acquisition  Workforce  Civilian  New  Hires 


DoD  ACQUISITION  WORKFORCE 
CIVILIAN  NEW  HIRES  (FY02-FY06) 


SPRDE 

Contracting 

Logistics 

PM 

BCEFM 

QA 

T&E 

IT 

FE 

Auditing 
Purchasing 
Other 
Mtg  Prod 
Property 


DAWIA 
Career  Field 

Number 
of  Hires 

Percentage 
of  All  Hires 

Percentage 
of  Career 
Field 

SPRDE 

7140 

35% 

22% 

Contracting 

4045 

20% 

17% 

Logistics 

2083 

10% 

18% 

PM 

1338 

6% 

16% 

BCEFM 

1007 

5% 

14% 

QA 

922 

4% 

13% 

T&E 

1455 

7% 

26% 

IT 

701 

3% 

16% 

FE 

542 

3% 

14% 

Auditing 

980 

5% 

28% 

Purchasing 

131 

1% 

8% 

Other 

157 

1% 

11% 

Mtg  Prod 

116 

1% 

9% 

Total 

20,680 

100% 

18% 

8,000 


NOTE:  Systems  Planning,  Research,  Development  and  Engineering  (SPRDE)  has  two  career  fields:  Science  and  Technology  Managers 
and  Systems  Engineering 


Another  specific  recommendation  in  response  to  the  anticipated  shortages  of  acquisition 
workforce  expertise  has  been  to  recruit  “the  best  qualified  technical  leaders  and 
specialists  from  the  private  sector  at  the  mid-career  and  senior  levels”  under  the 
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establishment  of  an  omnibus  legislative  initiative  (Defense  Science  Board,  2006:13). 
However,  because  this  projected  mass  retirement  and  resulting  experience  loss  plagues 
the  entire  national  workforce,  “competition  between  government  and  industry  for  new 
hires  will  intensify”  (Office  of  the  Secretary  of  Defense,  2007:13).  Nevertheless,  the 
entrance  of  new  mid-level  managers  into  the  acquisition  career  field  is  already  a  reality 
for  active-duty  officers  and  mid-level  government  civilians  cross-training  or  career¬ 
broadening  for  career  advancement  or  force-shaping  reasons.  However,  the  learning 
curve  for  overcoming  the  complexities  of  the  DoD  acquisition  environment  at  the  mid-to- 
upper  levels  of  responsibility  can  be  particularly  steep.  Furthermore,  time  and 
opportunities  for  traditional  on-the-job  training  (OJT)  or  formal  classroom  training  are 
difficult  due  to  increasing  operational  demands  yet  decreasing  personnel  numbers  and 
budgets. 

Two  initiatives  to  overcome  such  obstacles  and  better  meet  the  individual  training 
needs  of  its  diverse  workforce  have  been  the  recent  deployment  of  Defense  Acquisition 
University’s  “Core  Plus”  framework  and  revised  “Just-in-Time  (JIT)  Assignment- 
Specific  Training”  strategies  (Department  of  Defense  Acquisition,  Technology  and 
Logistics,  2007a:  26).  Core  Plus  strives  to  address  the  AT&L  Human  Capital  Strategic 
Plan  Workforce  Goal  #4:  “provide  learning  assets  at  the  point  of  need  to  support  mission¬ 
responsiveness  human  capital  development”  (Department  of  Defense  Acquisition, 
Technology  and  Logistics,  2007a:  26).  Being  both  performance-focused  and 
competency-based,  Core  Plus  “integrates  improved  competency  management  and  the 
DoD  AT&L  Performance  Learning  Model  (PLM)”  with  foundational  core  training  and 


18 


“additional  training  based  on  organization,  job  specific,  and  individual  professional 
development  needs”  (Department  of  Defense  Acquisition,  Technology  and  Logistics, 
2007a:  26).  Both  Core  Plus  and  JIT  training  are  marked  progress  towards  eliminating 
wasted  or  irrelevant  content — either  for  the  wrong  employees  or  at  the  wrong  times — and 
improving  competency-based  models  of  training.  However,  the  models  are  only  as  useful 
as  the  quality  of  the  competencies  identified  and  defined  within  them. 

Competency  Modeling 

As  part  of  the  Office  of  Personnel  Management  (OPM)  Human  Capital 
Assessment  and  Accountability  Framework  for  tracking  agency  performance  and 
effectiveness,  the  DoD  acted  on  the  2006  Quadrennial  Defense  Review  (QDR) 
recommendation  for  the  AT&L  Human  Capital  Strategic  Plan  to  be  competency-focused 
and  performance-based  (Hausmann  and  Tregar,  2006: 16).  A  joint  competency 
management  initiative  was  deployed  under  the  leadership  of  Defense  Acquisition 
University  (DAU)  and  with  the  expertise  of  analysts  from  the  Center  for  Naval  Analysis 
(CNA)  specializing  in  competency  modeling  (Office  of  the  Secretary  of  Defense, 

2007: 12).  This  partnership,  along  with  inputs  from  the  Federal  Acquisition  Institute 
(FAI),  should  enable  both  civilian  and  military  AT&L  employees  to  “work  towards  the 
same  group  of  competencies  and  have  greater  flexibility  and  mobility  to  acquisition  jobs 
throughout  the  Federal  workforce”  (Hausmann  and  Tregar,  2006: 16  ).  By  applying  best 
practices  of  competency  modeling  which  have  evolved  over  the  last  three  decades,  a 
standard  competency  model  for  each  career  field  in  the  AT&L  workforce  is  under 
development. 
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Historical  Evolution. 


The  modem  concept  of  competency  modeling  stems  from  the  work  of  social 
psychologist  David  McClelland,  who  opposed  the  practice  of  traditional  intelligence  tests 
as  a  means  for  measuring  aptitude  and  performance  as  outlined  in  his  1973  paper 
“Testing  for  Competence  Rather  than  Intelligence”  (Gannan  and  Johnson,  2006:  13). 
McClelland’s  “proposed  competencies — outcome -relevant  measures  of  knowledge,  skill, 
abilities,  and  traits  and/or  motives”  became  widely  accepted  especially  through  the  1990s 
as  organizations  were  facing  rapid  workforce  changes  and  needing  a  more  flexible 
framework  for  managing  workforce  requirements  (Garman  and  Johnson,  2006:  13). 
Organizations  began  using  competencies  to  identify  high-performing  outstanding 
employees,  becoming  the  basis  for  recruitment,  selection,  and  development  strategies  for 
the  greatest  return  on  human  capital  investment  (Rodriguez,  Patel,  Bright,  Gregory,  and 
Gowing,  2002:  310).  The  federal  government  was  also  experiencing  similar  workforce 
challenges,  so  the  Office  of  Personnel  Management  (OPM)  began  integrating  elements  of 
competency  models  in  1990,  recognizing  the  potential  for  wide-spread  application  across 
the  federal  government  and  in  particular  its  human  resource  management  challenges 
(Rodriguez  and  others,  2002:  310).  However,  many  industries  still  struggled  with 
ambiguities  and  inconsistencies  within  this  fledgling  model. 

In  1997,  the  Professional  Practice  Committee  and  the  Scientific  Affairs 
Committee  of  the  Society  for  Industrial  and  Organizational  Psychology  conducted  a  two- 
year  investigation  into  the  antecedents  of  competency  modeling  and  examined  the  current 
range  of  practices  to  date  (Shippmann  and  others,  2000:  703).  The  report  compared  and 
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contrasted  competency  modeling  to  job  analysis,  presenting  strengths  and  weaknesses  of 
each  as  they  existed  then,  and  recommended  how  practitioners  and  researchers  could  use 
a  combined  conceptual  framework  to  guide  future  efforts  towards  developing  standards 
for  practice.  It  also  identified  and  defined  ten  variables  for  effective  modeling 
methodologies  based  on  levels  of  rigor  required  (Shippmann  and  others,  2000:713).  The 
authors  suggested  that  the  practice  of  competency  modeling  could  possibly  extend 
beyond  the  traditional — albeit  rigorous — job  analysis  methodologies  to  date,  simply  due 
to  its  strength  in  identifying  commonalities  vice  differences  “across  jobs,  job  groups, 
occupational  groups,  business  segments,  and  so  forth,  in  an  effort  to  build  platforms  of 
information. .  .used  to  support  a  broad  range  of  applications”  (Shippmann  and  others, 
2000:733). 

The  Office  of  Personnel  Management  (OPM)  recognized  the  value  of  both  and 
“sought  to  incorporate  traditional  job  analysis  methodology  into  the  development  of 
competency  models  to  provide  an  empirical  foundation  for  the  use  of  competencies  by 
employees,  managers,  and  human  resource  (HR)  professionals”  (Rodriguez  and  others, 
2002:  310).  In  the  late  1990s,  OPM  initiated  what  would  become  a  3-part  study  to 
consolidate  information  about  competencies  in  the  context  of  “emerging  structures,” 
changing  roles,  and  various  competency  models  (Office  of  Personnel  Management, 

1999: 1).  There  were  several  trends  the  study  sought  to  take  into  account:  organization 
restructuring,  downsizing  of  the  HR  workforce  and  delegating  of  HR  authority,  and 
influx  of  new  technologies  (Office  of  Personnel  Management,  1999: 1).  One  issue  that 
the  OPM’s  study  helped  clarify  for  puzzled  practitioners  was  how  competencies  related 
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to  traditionally  binned  knowledge,  skills,  and  abilities  (KSAs).  The  study  explained  how 
KSAs  would  serve  as  the  foundation  for  competency  models  by  focusing  “typically  on 
what  is  needed  to  do  the  job  today,”  while  competencies  would  build  upon  those  same 
KSAs  but  also  include  traits,  motives,  and  behaviors  and  thus  “be  used  to  assess  and  train 
employees  for  future  needs”  (Office  of  Personnel  Management,  1999:9).  The  OPM  study 
also  suggested  that  clustered  sets  of  competencies  are  useful  in  detennining  “superior” 
versus  just  “basic”  performance  (Office  of  Personnel  Management,  1999:9). 

The  Federal  Acquisition  Institute  (FAI)  also  valued  how  the  “competency 
approach  provides  an  empirically  based  framework  that  focuses  on  the  full  range  of 
competencies  required  for  success  on  the  job”  (Federal  Acquisition  Institute,  2003:4). 
Building  on  the  guidance  outlined  by  OPM,  in  1999  the  FAI  initiated  a  competency- 
based  career  development  program  (Federal  Acquisition  Institute,  2003:2).  In  its  2003 
Report  on  General  Competencies  for  the  Federal  Acquisition  Workforce,  FAI  outlines  the 
methodology  and  results  of  conducting  a  competency-based  assessment  through  its 
piloting  of  the  Contract  Specialist  career  field  (Federal  Acquisition  Institute,  2003:4). 

The  report  recognizes  that  not  all  acquisition  professionals  may  be  required  to 
demonstrate  all  the  same  competencies,  nor  to  the  same  perfonnance  level;  rather, 
organizational  structure,  missions,  duties,  or  tasks  will  shape  which  competencies  are 
critical  and  at  what  level  (Federal  Acquisition  Institute,  2003:4).  This  same  approach  is 
enforced  in  the  Project  Management  Institute’s  Project  Manager  Competency 
Development  Framework,  which  explains  how  some  industries  or  organizations  may 
require  technical  competencies  specific  to  domain,  regulatory,  legal,  or  safety  standards 
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and  thus  should  supplement  their  framework  with  specific  competencies  to  meet  specific 
needs  (Project  Management  Institute,  2007:3).  Establishing  a  standardized  competency- 
based  framework  for  each  functional  area  will  allow  “leadership  of  the  acquisition 
community  [to]  provide  consistency  across  agencies,  across  acquisition  positions,  and 
across  perfonnance  management  efforts”  (Federal  Acquisition  Institute,  2003:5).  The 
benefits  extend  down  to  the  individual  level  as  well,  providing  the  “common  language  to 
communicate  about  job  requirements  to  potential  recruits,  preferred  course  content  to 
educational  institutions,  and  performance  expectations  and  career  development 
opportunities  to  employees,”  thus  potentially  motivating  employees  to  “stay  and  grow  in 
the  field”  (Federal  Acquisition  Institute,  2003:5).  By  better  communicating  position 
expectations,  “higher  quality  applicants  from  wider,  more  diverse  sources”  could  enable  a 
better  fit  between  employees  and  positions  and  thus  a  “more  direct  link  between  the  work 
and  agency  goals  and  budgets”  (Federal  Acquisition  Institute,  2003:5).  Other 
applications  include  rotational  assignments  or  career  broadening,  on-the-job  or  just-in- 
time  training,  and  performance  expectations  or  individual  development  plans  (IDPs) 
necessary  within  a  changing  workforce  enviromnent. 

Current  Practices. 

Both  competency  modeling  and  job  analysis  have  continued  to  evolve  and  are 
currently  integrated  within  the  Office  of  Personnel  Management  (OPM)  framework  for 
workforce  management  purposes.  OPM  defines  job  analysis  as  a  “systematic  procedure 
for  gathering,  documenting,  and  analyzing  information  about  the  content,  context,  and 
requirements  of  job. .  .to  develop  a  clear  understanding  of  the  tasks  performed  on  the  job, 
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as  well  as  the  competencies  needed  for  successful  performance”  (Office  of  Personnel 
Management,  2007:  3).  Consistent  with  other  definitions  now  accepted  across  the 
literature,  the  Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (OSD(AT&L))  defines  a  competency  as  an  “observable,  measurable  pattern  of 
knowledge,  skills,  abilities,  behaviors  and  other  personal  characteristics  that  an  individual 
needs  to  perform  work  roles  or  occupational  functions  successfully”  (Ahern  and 
Anderson,  2008:  1).  Thus,  the  definition  of  competency  model  is  “a  framework  that 
describes  the  full  range  of  competencies  required  to  be  successful  in  a  particular 
occupation”  (Office  of  Personnel  Management,  2007: 19).  In  addition  to  establishing 
behaviors  aligned  with  technical  competencies,  the  OPM  framework  also  includes 
general — or  professional — competencies  candidates  should  possess  (Office  of  Personnel 
Management,  2007: 19).  The  ability  to  provide  broader  flexibility  in  describing  desired 
capabilities  across  functional  areas  as  well  as  align  human  capital  management  with 
organizational  strategic  goals  is  the  overarching  strength  to  competency  models  (Prahalad 
and  Hamel,  1990:91).  This  premise  is  the  foundation  for  DoD’s  AT&L  Human  Capital 
Strategic  Plan,  which  aims  to  assist  senior  leaders  in  implementing  workforce  strategies 
to  address  critical  skill  gaps  and  target  new  education  and  training  resources  to  meet  the 
strategic  challenges  of  the  future  (Office  of  the  Secretary  of  Defense  2007:  12). 

Chartered  with  this  mission,  the  Defense  Acquisition  University  (DAU)  sought 
assistance  from  the  Center  for  Naval  Analysis  (CNA)  with  developing  competency 
models  for  each  of  the  functional  areas  within  the  AT&L  workforce.  The  CNA  analysts 
relied  extensively  on  the  theory  and  methodology  advocated  in  The  Art  and  Science  of 
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Competency  Models  by  authors  Anntionette  D.  Lucia  and  Richard  Lepsinger  (Tregar, 
2008).  Lucia  and  Lepsinger  outline  four  workforce  management  systems  for  integrating 
competency  models:  selection  (hiring),  training  and  development,  performance 
appraisals,  and  succession  planning  (Lucia  and  Lepsinger,  1999: 1 14).  To  date,  the 
training  and  development  arena  is  the  primary  target  of  competency-based  initiatives 
within  DoD.  The  CNA  efforts  resulted  in  an  Interim  Competency  Model  for  the  AT&L 
program  management  career  field  (Hausmann  and  Tregar,  2006:4)  and  upon  which  this 
research  methodology  is  modeled.  The  AT&L  model — still  under  refinement — is  also 
serving  as  a  framework  for  pilot  programs  testing  competency-based  initiatives  for  other 
workforce  applications.  For  example,  from  2005-2007  the  Air  Force  Electronic  Systems 
Center  at  Flanscom  Air  Force  Base,  MA,  partnered  with  OPM  to  undergo  a  competency- 
based  assessment  process  for  employee  selection  as  part  of  a  workforce  selection  and 
succession  development  effort  (Office  of  Personnel  Management,  2007: 1).  Similar 
efforts  are  being  assessed  for  other  Air  Force  acquisition  organizations,  such  as  the  Air 
Armament  Center  at  Eglin  Air  Force  Base,  FL,  and  the  Air  Logistics  Center  at  Warner- 
Robins  Air  Force  Base,  GA  (Fliggins,  2007:  20). 

According  to  AT&L,  the  goal  is  to  develop  models  or  audits  of  all  career  fields  by 
October  2008,  with  sample  assessments  of  six  additional  career  fields  by  December  2008 
(Department  of  Defense  Acquisition,  Technology  and  Logistics  4th  Estate,  2007b:  9). 

The  intent  is  to  align  the  competencies  within  each  model  with  the  DAWIA  certification 
program,  DAU  courseware,  Human  Capital  Strategic  Plan  initiatives,  and  Individual 
Development  Plan  program  (Department  of  Defense  Acquisition,  Technology  and 
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Logistics  4th  Estate,  2007:  9).  The  methodology  outlined  in  the  2006  AT&L  report, 
Program  Management  Career  Field:  Interim  Competency  Model,  serves  as  the  basis  on 
which  this  research  is  patterned  as  detailed  in  Chapter  III:  Methodology. 
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ITT.  Methodology 


The  methodology  for  this  research  is  patterned  on  a  2006  study  conducted  by 
analysts  from  the  Center  for  Naval  Analysis  (CNA)  and  Defense  Acquisition  University 
(DAU)  on  behalf  of  the  Department  of  Defense  (DoD)  Acquisition,  Technology  and 
Logistics  (AT&L)  leadership  and  workforce.  The  initial  effort  is  outlined  by  Hausmann 
and  Tregar  (2006)  in  their  review.  Improving  the  Certification,  Training,  and 
Development  of  the  AT&L  Workforce:  Program  Management  Career  Field  Interim 
Competency  Model.  This  chapter  outlines  the  data  collection  and  analysis  process  of  the 
original  AT&L  study,  the  data  collection  process  for  this  research  study,  and  the  analysis 
process  for  comparing  the  results  of  the  two  studies. 


AT&L  Interim  PM  Competency  Model 

The  AT&L  study  follows  a  competency  model  methodology  prescribed  across  the 
literature  from  both  private  and  public  sectors  (Hausmann  and  Tregar,  2006:5).  It  mirrors 
a  common  development  strategy  espoused  in  academic  and  business  literature  which 
follows  a  six-stage  process  (Boulter,  Dalziel,  and  Hill,  1998:32): 

1 .  Performance  criteria:  define  the  criteria  for  superior  performance  in  the  role 

2.  Criterion  sample:  choose  a  sample  of  people  performing  the  role  for  data 
collection 

3.  Data  collection:  collect  sample  data  about  behaviors  that  lead  to  success 

4.  Data  analysis:  develop  hypothesis  about  the  competencies  of  outstanding 
performers  and  how  these  competencies  work  together  to  produce  desired 
results 

5.  Validation:  validate  the  results  of  data  collection  and  analysis 

6.  Application:  apply  the  competency  models  in  human  resource  activities  as 
needed 
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The  AT&L  model  compresses  these  stages  into  a  three-step  qualitative  approach:  first, 
convene  an  expert  panel  to  establish  the  competency  framework  and  identify  subject 
matter  experts  (SMEs);  second,  use  “key  situation  structured  interviews”  with  the  SMEs 
to  validate  and  expand  on  the  competencies  based  on  specific  examples  of  behavior, 
followed  by  specific  questions  rating  the  competencies  to  compute  criticality  rankings; 
third,  conduct  a  validation  survey  of  the  hypothesized  competencies  in  order  to  generalize 
or  make  inferences  about  competency-based  initiatives  for  the  workforce  at  large 
(Hausmann  and  Tregar,  2006:6).  At  the  time  of  its  2006  interim  report,  the  AT&L  study 
had  completed  the  first  two  steps;  the  third  step — the  final  validation  survey — is  currently 
underway  (Tregar,  2008). 

Step  1:  Expert  Panel. 

This  first  stage  of  the  data  collection  process  involved  gathering  highly- 
knowledgeable  functional  leaders  to  develop  the  competency  model  framework  and 
identify  subject  matter  experts  (SMEs)  from  their  career  field.  The  focus  group,  with  the 
assistance  of  Defense  Acquisition  University  (DAU)  guidance  and  past  PM  competency 
model  reviews,  proposed  legacy  program  management  competencies  which  would 
become  the  baseline  the  SMEs  would  later  expand  upon  and  rate  (Tregar,  2008).  The 
expert  panel  identified  SMEs  with  more  than  two  years  of  experience  who  were 
recognized  as  “superior  performers”  from  within  their  career  field  and  having  the  ability 
to  clearly  communicate  examples  of  competencies  required  in  the  job  (Hausmann  and 
Tregar,  2006:25).  The  AT&L  study’s  expert  panel  consisted  of  participants  from  across 
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the  federal  government — both  Department  of  Defense  (DoD)  and  civilian  agencies 
(Hausmann  and  Tregar,  2006:26). 

Step  2:  SME  Data  Collection. 

With  the  baseline  competencies  and  framework  established,  the  Subject  Matter 
Expert  (SME)  data  collection  process  used  in-person  interviews  via  focus  groups  along 
with  electronic  online  tools  (Hausmann  and  Tregar,  2006:27).  The  SMEs  were  multi¬ 
service,  serving  at  Wright  Patterson  Air  Force  Base  OH,  Naval  Air  Station  Patuxent 
River  MD,  and  Army  Redstone  Arsenal  in  Huntsville  AL  (Hausmann  and  Tregar, 
2006:3 1).  A  total  of  70  SMEs  participated  (with  usable  data  from  69)  representing  the 
following  demographics  (Hausmann  and  Tregar,  2006:34): 

•  78.3%  were  from  DoD;  2 1 .7%  were  from  a  Civilian  Agency 

•  77. 1%  were  from  the  Program  Management  career  field  (the  others  represented 
information  technology,  systems  engineering,  business  financial  management, 
manufacturing,  and  contracting  career  fields) 

•  72.9%  had  more  than  12  years  of  experience;  15.7%  had  7-12  years  of 
experience;  8.6%  had  3-6  years  of  experience;  2.9%  had  less  than  2  years  of 
experience 

•  86.8%  were  certified  at  the  Senior/Expert  level;  13.2%  at  the  Journeyman  level; 
and  0%  at  the  Entry  level 

While  this  demographic  data  show  the  majority  of  SMEs  as  experienced,  top-certified 
DoD  program  managers  and  thus  reinforcing  the  credibility  of  their  inputs  for  the 
intended  outputs,  the  researchers  acknowledge  that  the  lack  of  proportional  workforce 
representation  limits  the  study’s  ability  to  generalize  sample  results  to  the  workforce  at 
large — hence  the  need  for  the  final  validation  survey  (Hausmann  and  Tregar,  2006:35). 
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Nevertheless,  the  researchers  felt  the  consistency  in  the  responses  was  adequate  for  use  as 
an  interim  competency  model  for  program  management  professional  development 
purposes  (Hausmann  and  Tregar,  2006:35). 

In  obtaining  the  SME  inputs  via  the  in-person  interviews,  the  researchers  followed 
a  methodology  known  as  “key  situation”  or  “critical  incident  technique,”  during  which 
SMEs  are  asked  to  “describe  an  effective  situation  or  experience  when  they  felt 
particularly  effective  and  confident  on  the  job”  (Hausmann  and  Tregar,  2006:27)  and 
identify  key  actions  which  resulted  in  an  effective  outcome  (Hausmann  and  Tregar, 
2006:28).  As  a  means  to  quickly  identify  situations,  organize  thoughts,  and  write 
descriptively,  the  researchers  had  SMEs  follow  the  “STARR”  process:  ‘Situation/Task’ 
(explain  the  situation  context,  actions,  work  effort);  ‘Action’  (outline  steps  taken  toward 
an  effective  outcome);  ‘Reasoning’  (explain  rationale  that  led  to  the  actions);  and 
‘Results’  (present  outcomes  of  the  key  situation)  (Hausmann  and  Tregar,  2006:28).  After 
describing  the  PM  situations  in  detail,  the  SMEs  listed  which  of  the  pre-identified  set  of 
technical  competencies  were  associated  with  each  event.  Additionally,  the  SMEs  rated 
professional  competencies  with  respect  to  the  situations  from  those  listed  in  the  Office  of 
Personnel  Management  (OPM)  Executive  Core  Qualification  standard  (Hausmann  and 
Tregar,  2006:28).  The  SME’s  interviews  and  ratings  align  with  literature  regarding  the 
importance  of  including  both  professional  and  technical  competencies  in  a  competency 
model:  “effective  PMs  share  a  common  foundation  of  basic  knowledge  and  skills;  top 
performing  PMs  also  exhibit  key  leadership  behaviors  that  allow  them  to  employ  these 
building  blocks  more  effectively  to  achieve  superior  results”  (Gadekin,  2005:1 1). 
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Based  on  the  SME  interviews,  the  AT&L  research  team  used  a  qualitative  content 
analysis  to  refine  the  competency  framework  with  specific  behavior-based  descriptions 
for  each  competency  definition  (Hausmann  and  Tregar,  2006:32).  The  SMEs  then  rated 
the  competencies  based  on  their  perceptions  of  importance,  frequency,  and  career  levels 
when  first  used  per  the  following  questions  and  Likert  scale  responses  (Hausmann  and 
Tregar,  2006:29): 

•  Importance:  “What  is  the  degree  of  impact  this  work  function  has  on  job 
performance?”  (1-Not  Important;  2-Somewhat  Important;  3-Important;  4-Very 
Important;  5 -Extremely  Important) 

•  Frequency:  “How  often  is  this  competency  used  in  performing  work?  (1 -Never; 
2-Sometimes;  3-Often;  4-Frequently;  5-Very  Frequently) 

•  Level  First  Used:  “At  what  point  in  your  career  did  you  FIRST  use  this  work 
function  to  perfonn  your  job?”  (1-Entry;  2-Journeyman;  3-Senior/Expert) 

The  research  team  then  computed  a  “criticality  score”  for  each  competency  based  on  the 

group  average  of  the  importance  and  frequency  means.  These  criticality  scores  suggest  a 

prioritized  ranking  of  the  competencies  most  critical  to  successful  program  management 

and  at  levels  first  needed,  which  becomes  the  basis  of  comparison  for  this  research  study 

regarding  critical  competencies  for  managing  Science  and  Technology  programs. 

S&T  PM  Competency  Study 

Because  the  scope  of  this  research  is  to  analyze  program  management 
competencies — but  within  the  context  of  Department  of  Defense  (DoD)  Science  and 
Technology  (S&T)  programs — the  AT&L  PM  study  serves  as  a  valid  model  from  which 
to  start.  Although  “starting  from  scratch  is  appropriate  for  developing  a  competency 
model  [...]  to  yield  role-and  company-specific  results,”  it  is  more  time-consuming  than 
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the  alternative  approach  of  starting  with  a  validated  model  which  is  “best  suited  for 
leadership  and  management  roles  that  cut  across  several  functions”  (Lucia  and  Lepsinger, 
1999:53).  Thus,  a  content  analysis  of  the  competencies  identified  for  two  Science  and 
Technology  Manager  (STM)  courses  presented  in  Appendix  C  (Department  of  Defense, 
1999: 13 1)  and  those  from  the  AT&L  PM  model  suggests  initial  face  validity  exists  for 
using  the  same  competency  framework. 

Step  1:  Expert  Panel. 

With  the  competency  model  framework  predefined  by  the  AT&L  study,  this  step 
in  the  research  methodology  provides  an  opportunity  to  consult  with  functional  S&T  and 
PM  leaders  for  insights  on  historical  and  organizational  background  infonnation  relevant 
to  the  scope  of  this  study,  and  to  identify  potential  S&T  program  management  SMEs  for 
data  collection.  The  expert  panel  members  provide  background  infonnation  and  guidance 
into  issues,  questions,  or  concerns  that  could  possibly  shape  the  SME  interviews  and 
future  data  analysis.  This  step  also  alerts  stakeholders  of  the  research  goals  and  potential 
benefits,  in  this  case  gamering  research  sponsorship  from  Headquarters  Air  Force 
Materiel  Command’s  Requirements  Directorate  and  Technology  Transition  Division, 
along  with  organizational  endorsement  by  the  Air  Force  Research  Laboratory  Directorate 
of  Personnel.  With  a  list  of  SMEs  identified  based  on  the  same  criteria  as  the  AT&L 
study — at  least  two  years  of  experience  managing  S&T  programs  and  considered  a 
“superior  performer” —  an  invitation  could  be  extended  to  SMEs  to  voluntarily 
participate  in  the  research  study. 
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Step  2:  SME  Data  Collection. 

Upon  meeting  the  criteria  for  an  Institutional  Review  Board  exemption  (Appendix 
A),  the  researcher  electronically  mails  an  invitation  to  all  SMEs  requesting  their 
voluntary  participation  in  this  study.  The  SMEs  then  select  their  preferred  method  of 
interface — in  person,  telephone,  or  email — for  providing  responses  to  a  structured 
interview  fonnat  consisting  of  four  parts  (Appendix  B): 

•  Part  I:  closed-ended  demographic  questions  regarding  years  of  experience, 
certifications,  level,  and  academic  degrees 

•  Part  II:  three  closed-ended  questions  regarding  the  importance,  frequency,  and 
levels  first  used  for  36  technical  competencies 

•  Part  III:  three  closed-ended  questions  regarding  the  importance,  frequency,  and 
levels  first  used  for  27  professional  competencies 

•  Part  IV :  three  open-ended  questions  regarding  any  competencies  missing  from  the 
existing  model,  the  usefulness  of  applying  a  competency  model  for 
hiring/evaluation  criteria,  and  the  criticality  of  technical  degrees  for  successful 
program  management  within  S&T 

A  pilot  test  of  the  structured  interview,  useful  for  assessing  how  new  procedures  or 
instruments  work  (Patten,  2005:55),  estimates  30  minutes  for  completion.  Actual  time 
varies  by  respondent  depending  on  how  much  discussion  transpires  during  the  interview, 
or  between  interruptions  for  those  responding  electronically. 

As  recommended  in  other  studies,  a  single  interviewer  conducts  the  structured 
interviews  in  order  to  “ensure  a  unifonn  approach  to  the  data  collection  process” 

(Greiner,  Dooley,  Shurik,  and  McNutt,  2002:125).  The  format  remains  consistent 
beginning  with  a  brief  discussion  explaining  the  purpose,  process,  and  bounds  of  the 
research  and  structured  interview.  Although  maintaining  the  same  competencies, 
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definitions,  and  measures  of  evaluation,  additional  demographic  data  and  three  open- 
ended  questions  are  added  to  the  S&T  PM  interview.  After  collecting  individual  data 
from  each  SME,  the  responses  are  consolidated  into  a  master  Excel  spreadsheet  and  the 
original  notes  destroyed.  No  identifiable  data  or  indicators  link  respondents  to  their 
responses. 

Data  Analysis 

With  the  data  compiled  into  a  master  Excel  spreadsheet,  the  results  can  be 
grouped,  computed,  and  analyzed.  A  quantitative  analysis  can  be  applied  to  the 
descriptive  statistics  of  the  demographic  data  and  competency  ratings,  with  a  qualitative 
analysis  applied  to  the  open-ended  responses. 

Part  I:  Demographic  Data. 

The  first  part  of  the  structured  interview  consists  of  self-reported  demographic 
data:  the  participants’  experience  based  on  years  managing  S&T  programs  or  supervising 
those  who  do;  expertise  based  on  professional  certifications  and  skill  levels;  education 
backgrounds  based  on  degrees  awarded;  and  organizational  influences  based  on  years 
serving  in  each  of  the  Air  Force  Research  Laboratory  (AFRL)  organizations.  The  data  is 
categorized  then  reported  in  terms  of  frequency  and  percentages  in  order  to  describe  the 
respondents  to  the  research  audience. 

Parts  II  &  III:  Competency  Criticality  Means. 

The  second  and  third  parts  of  the  structured  interview  consist  of  closed-ended 
questions  for  each  technical  and  professional  competency  defined  in  the  AT&L  study. 
Each  respondent’s  numerical  answers  are  based  on  a  5-point  Likert-scale  for  the  first  two 
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questions  about  a  competency’s  importance  and  frequency  of  use.  The  sample  mean  for 
each  of  the  first  two  ratings  are  then  computed  by  summing  all  respondents’  numerical 
ratings  for  each  question  then  dividing  by  the  sample  size.  Once  the  means  for  a 
competency’s  importance  and  frequency  ratings  are  computed,  the  criticality  score  for 
each  competency  is  computed  by  averaging  the  two  means:  [(  X ;  +  X  f)/2].  Based  on  the 
criticality  scores  of  competencies  for  both  sample  groups,  the  research  can  compare  the 
descriptive  statistics  to  assess  differences  in  prioritized  rankings.  This  analysis  addresses 
the  investigative  question  of  whether  differences  exist  in  performance  competencies — 
either  more  or  less  critical — between  traditional  acquisition  program  managers  and  those 
managing  Science  and  Technology  programs. 

The  next  investigative  question — whether  differences  between  the  criticality 
means  are  statistically  significant — requires  the  use  of  inferential  statistics,  which  “help 
draw  references  about  the  effects  of  sampling  errors  on  the  results  that  are  described  with 
descriptive  statistics”  (Patten,  2005:97).  The  statistical  hypothesis  test  is  structured  as 
follows: 

•  H0:  There  is  no  statistically  significant  difference  between  the  means,  translated  as 
‘there  is  no  true  difference  between  the  criticality  scores  of  competencies  for 
Acquisition  PMs  and  S&T  PMs,  other  than  that  which  is  created  merely  by 
chance  due  to  sampling  error’ 

•  Ha :  The  difference  between  the  means  is  statistically  significant. . .  .translated  as 
‘the  differences  between  the  criticality  scores  of  competencies  for  ACQ  PMs  and 
S&T  PMs  is  due  to  more  than  mere  sampling  error’ 
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Before  proceeding  to  a  null-hypothesis  statistical  significance  test,  or  the  Student’s  t- 
distribution  test,  the  following  conditions  must  be  assessed  (McClave,  Benson,  Sincich, 
2005:486): 

•  independence  or  dependence  of  the  samples 

•  random  or  not  random  selection  of  the  samples 

•  nonnal  or  abnormal  distribution  of  the  data 

•  equality  or  inequality  of  the  variances 

This  study  can  assume  independence  and  acknowledge  the  limitations  of  the  purposive, 
non-random  sampling.  However,  the  latter  two  assumptions  require  further  analysis. 

Regarding  the  normality  of  the  data  distribution,  several  types  of  descriptive 
methods  can  be  used  to  check  for  nonnality.  Graphical  techniques  such  as  a  histogram  or 
stem-and-leaf  display  will  suggest  a  visual  representation  of  the  shape  of  the  curve,  which 
for  normality  should  be  mound-shaped  and  symmetric  about  the  mean  (McClave, 

Benson,  and  Sincich,  2005:285).  Another  graphical  test  of  normality  is  the  Q-Q  plot, 
which  presents  the  observed  values  as  dots  plotted  along  a  straight  diagonal  line  of 
expected  values  for  a  normally  distributed  data  set  (Field,  2005:96).  However,  to  balance 
the  subjective  analysis  of  graphical  displays,  an  objective  mathematical  test  for  normality 
is  the  Shapiro-Wilk  test  which  compares  the  scores  in  the  sample  to  a  normally 
distributed  set  of  scores  with  the  same  mean  and  standard  deviation  (Field,  2005:93).  If 
the  test  is  statistically  significant  (p  <.05) — as  computed  through  a  statistical  software 
package — then  the  distribution  is  significantly  different  from  a  normal  distribution  (Field, 
2005:93).  However,  a  drawback  to  using  this  mathematical  analysis  exclusively  is  that 
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the  larger  the  sample  size,  the  more  the  tendency  to  get  significant  results  from  small 
deviations  in  normality — thus  masking  whether  the  deviation  from  normality  is  enough  to 
bias  statistical  procedures  (Field,  2005:93).  Therefore,  this  research  applies  all  four 
methods  to  conservatively  assess  the  distribution  normality  of  the  sample.  It  should  be 
noted  that  without  the  original  AT&L  study  data,  similar  assessment  of  normality  is  not 
possible;  however,  because  the  sample  sizes  are  both  considered  large  [(ni=65-69)  >  30 
and  (n2=41)  >  30],  the  Central  Limit  Theorem  “guarantees  that  the  sampling  distribution 
of  ( X  i  -  X  2)  will  be  approximately  normal  regardless  of  the  shapes  of  the  underlying 
probability  distributions  of  the  populations  (McClave,  Benson,  and  Sincich,  2005:483). 

The  final  consideration  for  use  of  the  t-test  is  the  assumption  of  equality  of  the 
variances.  Looking  at  the  standard  deviations  of  both  samples,  they  clearly  are  not  the 
same.  However,  a  more  rigorous  statistical  test  of  this  assumption  would  be  the  2-tailed 
F-distribution  test  based  on  inferences  about  the  ratio  of  the  two  variances,  and  applicable 
when  both  samples  are  random,  independent,  and  normally  distributed  (McClave, 

Benson,  Sincich,  2005:528).  The  outcome  of  the  F-test — whether  or  not  to  reject  the  null 
hypothesis  that  the  variances  are  equal — will  drive  which  computation  for  degrees  of 
freedom  (v)  to  use  in  combination  with  the  computed  test-statistic  for  analyzing  the 
probability  (p-value)  of  statistical  significance  (McClave,  Benson,  Sincich,  2005:524). 
Upon  addressing  independence,  randomness,  normality,  and  unequal  variance  of  the  two 
unequal  sample  sizes,  the  Satterhwaite’s  approximation  /  Welch’s  adaptation  for  test 
statistic  (0  and  degrees  of  freedom  (v)  apply  as  shown  in  Figure  1  (McClave,  Benson, 
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Sincich,  2005:492).  If  the  resulting  /7-value  is  <.05,  the  difference  in  the  means  for  that 
competency  is  considered  statistically  significant,  thus  rejecting  the  H0  hypothesis. 


t  =  (X1-X2)  I  /ill  +  H 

/  A/  ni  n2 

t  is  based  on  degrees  of  freedom  equal  to : 

(s-i2/rii  +  s22/n2)2 

(Sl2/m)2  |  (s22/n2)2 

ni  -  1  n2  -  1 


where: 

x=  mean 

s2=  sample  variance 
n  =  sample  size 


Figure  1:  Satterhwaite  /  Welch  Equations 


Part  IV:  Open-Ended  Questions. 

The  final  part  of  the  study  involves  a  qualitative  analysis  of  the  responses  to  the 
three  open-ended  questions  based  on  a  grounded  theory  approach.  As  an  inductive 
method  of  analysis,  it  should  lead  to  the  emergence  of  theories  through  consideration  and 
analysis  of  the  data  (Patten,  2005: 153).  The  responses  are  examined  for  “distinct, 
separate  segments  (such  as  ideas  or  experiences  of  the  participants)  and  are  ‘coded’  by 
identifying  them  and  giving  each  type  a  name”  (Patten,  2005: 153).  The  research  is 
intended  to  identify  themes  to  elucidate  the  opinions  of  the  S&T  program  managers 
regarding  competency  modeling  and  its  potential  applications  to  current  or  future  human 
resource  management  issues. 
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Constraints  &  Limitations 


The  methodology  used  in  this  study  has  some  natural  and  research-induced 
constraints  which  could  limit  its  broader  applications,  or  generalizations,  to  the  S&T  / 
R&D  program  management  community  at  large. 

Data  Collection. 

The  primary  constraint  involves  sampling  issues.  In  addition  to  sample  size 
limitations,  the  participants  are  not  randomly  selected;  rather,  both  studies  use  purposive 
sampling — specifically  expert  sampling — because  targeting  subject  matter  expert 
opinions  is  the  primary  aim  versus  proportionality.  Additional  sampling  bias  could  result 
from  the  element  of  volunteerism  by  respondents  since  “volunteers  may  be 
fundamentally  different  from  non-volunteers,”  as  well  as  the  element  of  convenience 
sampling — primarily  AFRL  SMEs  located  at  Wright-Patterson  AFB  (Patten,  2005:45). 

As  such,  the  demographic  data  of  the  participants  is  not  necessarily  representative  of  the 
broader  AT&L  community’s  S&T  program  managers.  Another  potential  constraint  could 
involve  researcher/response  bias,  where  the  interviewer  interaction  might  affect  the 
outcomes  through  clarifications  or  the  influence  of  social  desirability. 

Data  Analysis. 

The  issue  of  construct  validity  is  at  the  crux  of  the  study.  Upon  initial  inspection, 
the  AT&L  PM  competency  model  appears  to  have  face  validity — that  is,  it  appears  to  be 
a  valid  instrument  applicable  to  S&T  program  managers  (Patten,  2005:61).  By 
proceeding  through  the  interview  process,  the  research  tests  the  actual  content  validity — 
that  is,  whether  the  competencies  defined  within  the  competency  model  are  appropriate 
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for  what  is  intended  to  be  measured  and  analyzed  (Patten,  2005:68).  If  issues  with 
content  validity  emerge,  they  could  flaw  the  results  through  systematic  error,  thus 
bringing  the  reliability  of  the  criticality  scores  into  question. 

Nevertheless,  for  the  purpose  of  this  research  scope  and  for  comparability 
between  studies,  the  sample  participants  and  consistent  methods  support  the  intent  of  the 
research.  Any  observations  to  be  discovered  become  opportunities  for  further  research 
and  refinement  of  this  exploratory  topic,  ultimately  leading  to  a  reliable  competency 
model  for  aiding  successful  S&T  program  management  perfonnance. 
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IV.  Results  &  Analysis 


This  chapter  summarizes  the  data  collected  from  the  Science  and  Technology 
(S&T)  program  managers  (PMs)  and  analyzes  it  in  comparison  with  results  from  the 
AT&L  interim  report  Improving  the  Certification,  Training,  and  Development  of  the 
AT&L  Workforce.  It  first  presents  descriptive  statistics  on  the  demographic  data  of  the 
respondents.  Next  it  presents  descriptive  statistics  on  the  S&T  PM  responses  for  the 
technical  and  professional  competencies  studied,  along  with  inferential  statistics 
comparing  these  results  with  those  from  the  AT&L  study.  Lastly,  it  presents  the  results 
and  analysis  from  the  three  open-ended  interview  questions  unique  to  this  research  study. 

Part  I:  Demographic  Data 

Consultations  with  Expert  Panel  members  from  the  S&T  Program  Management 
community  at  the  Air  Force  Research  Laboratory  (AFRL)  resulted  in  the  identification  of 
85  Subject  Matter  Experts  (SMEs)  who  qualified  as  “superior  performers”  with  at  least 
more  than  two  years  of  experience  in  S&T  program  management.  Initially,  52 
volunteered  to  participate  in  the  study,  but  through  attrition  the  final  sample  size 
consisted  of  4 1  SMEs. 

Years  of  Experience. 

Experience  was  recorded  according  to  four  categories,  to  include  any  time  serving 
as  a  military,  civilian,  or  contracted  employee  in  S&T  program  management.  A  picture 
of  an  individual’s  full  career — not  just  current  position  status — was  valuable  in  gaining 
fuller  insight  into  the  types  of  positions  and  perspectives  that  might  be  shaping  a  SME’s 
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responses  to  the  interview  questions.  Of  the  41  respondents,  46%  have  served  as  active 
duty  military,  83%  have  served  as  a  civilian  employee,  20%  have  served  as  a  contracted 
employee,  and  44%  of  those  have  some  combination  of  experience  across  the  three 
categories.  The  number  of  years  served  within  each  type  of  position  status  is  broken  out 
in  Table  4. 


Table  4:  SME  Years  of  Experience 


*  44  %  Combined  Experience  (18) 


• 

39  %  >12  years 

46  %  Military  Experience  (19) 

• 

15.8%  >12  years 

• 

21.0%  7-12  years 

• 

31.6%  3-6  years 

• 

31.6%  <2  years 

83  % 

Civilian  Experience  (34) 

• 

47.1%  >  12  years 

• 

32.4  %  7-12  years 

• 

11.8%  3-6  years 

• 

8.8  %  <2  years 

20  % 

Contractor  experience  (8) 

• 

0.0  %  >  12  years 

• 

12.5  %  7-12  years 

• 

75.0%  3-6  years 

• 

12.5  %  <2  years 

(n  =  41); 

highlighted  respondents 

represent  combined  experience 


Years  of  Experience  in  Science  and  Technology  Program  Management 
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The  highest  category  of  service  represented  in  this  sample  group  is  civilian 
service  (83%),  with  47%  of  those  having  more  than  12  years  of  experience  managing 
S&T  programs  or  supervising  those  who  do.  Like  the  AT&L  study,  due  to  purposive 
SME  sampling,  employees  having  higher  levels  of  experience  dominate  the  sample  group 
with  disproportional  representation  of  employees  having  fewer  years  of  experience. 

An  additional  experience  factor  collected  was  the  number  of  years  respondents  have 
spent  across  the  different  Air  Force  Research  Laboratory  (AFRL)  Technology 
Directorates  (TDs) — providing  insight  into  potential  organizational  influencers  shaping 
responses.  They  are  broken  out  by  percent  across  the  TDs,  Headquarters  Plans  & 
Programs  (XP)  office,  or  “other”  S&T  organization  as  shown  in  Figure  2.  It  should  be 
noted,  however,  that  due  to  the  sampling  of  convenience  factor  required  for  conducting 
the  in-person  interviews,  the  respondents  are  mostly  representative  of  those  AFRL 
organizations  co-located  at  Wright  Patterson  AFB,  OH,  rather  than  all  of  AFRL’s  SME 
population  across  multiple  geographically  separated  locations. 
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Figure  2:  Organizational  Experience 


Professional  Certification  Levels. 

The  next  type  of  demographic  data  collected  includes  the  types  and  levels  of 
professional  certifications  held  by  the  S&T  PM  subject  matter  experts  (SMEs).  Per  the 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  the  AT&L  workforce  must 
acquire  certification  levels  based  on  years  of  experience  in  specifically  coded  positions, 
training,  and  education  in  accordance  with  standards  outlined  by  the  Acquisition 
Professional  Development  Program  (APDP).  The  APDP  certification  types  and  levels 
held  by  the  SMEs  interviewed  for  this  study  are  outlined  in  Figure  3. 
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Of  the  41  SMEs,  20  (49%)  held  certifications  in  more  than  one  APDP  area;  however,  38 
(93%)  held  Systems  Planning,  Research,  Development  and  Engineering  (SPRDE)  — 
Systems  Engineering  certifications,  but  only  19  (46%)  held  some  level  of  Program 
Management  certification.  The  specific  numbers  and  percentages  for  each  level  are 
broken  out  by  APDP  types  in  Table  5.  This  current  workforce  composition  suggests 
SPRDE  technical  development  supersedes  that  of  program  management  development  for 
this  AFRL  sample  group. 


Table  5:  APDP  Levels 


APDP  Type 

Level  I 

Level  II 

Level  III 

Program  Management 

5%  (2) 

22%  (9) 

22%  (9) 

SPRDE  -  SE 

5%  (2) 

12%  (5) 

76%  (31) 

Test  &  Evaluation 

12%  (5) 

10%  (4) 

15%  (6) 

Life  Cycle  Logistics 

5%  (2) 

2%  (1) 

0%  (0) 
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Another  observation  particular  to  the  AFRL  S&T  program  manager  model  is  that 
the  SPRDE  certifications  are  aligned  with  the  Systems  Engineering  subcategory  rather 
than  the  Science  &  Technology  Manager  (STM)  subcategory  of  the  AT&L  functional 
groups.  Therefore,  participants  were  asked  to  self-identify  their  STM  levels  according  to 
the  STM  course  competencies  outlined  in  the  DoD  Acquisition  Career  Management: 
Mandatory  Course  Fulfillment  Program  and  Competency  Standards  (Appendix  C)  and 
certification  level  descriptions  in  the  2007  Federal  Acquisition  Certification  for  Program 
and  Project  Managers  (Federal  Acquisition  Institute,  2007:5).  The  results,  as  presented 
in  Table  6,  suggest  that  of  the  41  AFRL  SMEs,  76%  are  considered  to  be  at  the  Expert 
Level,  24%  at  the  Journeyman  Level,  and  0%  at  the  Entry  Level.  Once  again,  the 
disproportionate  representation  is  recognized  as  a  natural  factor  of  purposive  SME 
sampling,  but  able  to  be  mitigated  through  the  use  of  a  final  validation  survey. 


Table  6:  STM  Levels 


APDP  Type 

Entry  /  Beginner 

Journeyman  /  Intermediate 

Expert  /  Advanced 

S&T  Management 

0%  (0) 

24%  (10) 

76%  (31) 

Academic  Degrees. 

The  last  type  of  demographic  data  collected  includes  the  types  of  academic 
degrees  held  by  the  Subject  Matter  Experts  interviewed.  The  respondents  were  asked  the 
number  of  engineering,  science  (chemistry,  physics,  biology),  math,  or  “other”  degrees 
held,  and  to  specify  degrees  in  the  “other”  category.  The  results  are  detailed  in  Figure  4. 
Academic  degrees  are  critical  qualifiers  for  each  of  the  AT&L  functional  career  fields. 
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The  current  standard  for  hiring  and  certifying  an  S&T  Program  Manager  requires  a 
technical  degree:  engineering,  science,  or  math.  However,  PMs  in  other  phases  of  the 
AT&L  lifecycle  do  not  always  have  technical  degrees;  rather,  they  usually  hold  business 
or  management  degrees.  Due  to  the  fluid  crossflow  of  military  assignments,  active  duty 
PMs  might  find  themselves  assigned  to  S&T  programs  with  or  without  a  technical 
degree.  However,  current  hiring  restrictions  prevent  even  the  most  superior-performing 
civilian  or  contracted  PMs  from  crossflowing  over  to  S&T  programs  without  a  technical 
degree. 


Figure  4:  Academic  Degrees 


Part  II:  Technical  Competencies 

The  second  part  of  this  study  examines  technical  competencies,  addressing  the 
research  question  of  whether  certain  competencies  are  more  or  less  critical  to  different 
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program  manager  groups.  In  the  study,  the  S&T  PMs  were  asked  to  review  the  AT&L 
PM  competencies  and  their  definitions,  then  provide  responses  to  the  same  questions 
about  each  of  the  competencies  (Appendix  B).  The  first  question  asked  the  respondents 
to  rate  the  degree  of  impact — in  terms  of  importance — each  competency  has  on  job 
performance,  using  a  5 -point  Likert  scale.  The  second  question  asked  the  respondents  to 
rate  how  often  each  competency  is  used  in  perfonning  work  in  tenns  of  frequency,  also 
using  a  5-point  Likert  scale.  The  average  of  these  two  means  was  computed  into  a 
“criticality  score”  for  each  competency,  enabling  a  rank  ordering  of  prioritized 
competencies  according  to  each  sample  group  as  shown  in  Table  7:  Comparison  of 
Technical  Competencies.  Although  beyond  the  scope  of  this  research  study,  the 
underlying  assumption  of  the  ranked  competencies  is  that  the  higher  a  competency  is 
ranked,  the  greater  effect  it  has  on  PM  performance  which  would  theoretically  show  up  in 
an  empirical  study  as  having  the  largest/larger  regression  coefficient  (j:S)  than  lower 
ranked  competencies. 
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Table  7:  Comparison  of  Technical  Competencies 


TECHNICAL  COMPENTENCY  TITLES 

S&T  Results 

AT&L  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

TC  8 

Communications  Management  &  1PT/1PPT  Process 

41 

0.79 

4.34 

1 

1 

4.39 

0.71 

69 

.74814 

TC20 

Transition  Techniques 

41 

0.83 

3.88 

2 

33 

3.04 

0.94 

68 

.00000** 

TC28 

Contract  Approach,  Requirements  &  Supporting  Documents, 
Prepare  &  Issue  Solicitation 

41 

1.01 

3.80 

3 

7 

4.08 

0.75 

68 

.13427 

TC13 

Technical  Management  Process 

41 

0.86 

3.79 

4 

5 

4.13 

0.64 

68 

.03305* 

TC29 

Source  Selection,  Contract  Award,  Contract  Administration, 

&  Contract  Closeout 

41 

0.84 

3.73 

5 

9 

3.99 

0.79 

69 

.11626 

TC14 

Technical  Process 

41 

1.03 

3.61 

6 

12 

3.95 

0.71 

68 

.06772 

TC  1 

Requirements  Process 

41 

0.88 

3.55 

7 

3 

4.28 

0.64 

68 

.00002** 

TC19 

S&T  Goal,  Program  Considerations,  &  Technology 
Engineering 

41 

1.08 

3.55 

7 

31 

3.13 

0.88 

68 

.03908* 

TC33 

Financial  Reporting  &  Oversight,  Dpt/ Agency  Programming, 
Planning,  &  Budgeting  System 

41 

1.04 

3.55 

7 

9 

3.99 

0.78 

68 

.02202* 

TC  5 

Risk  Management 

41 

0.86 

3.44 

10 

4 

4.17 

0.75 

69 

TC  3 

Core  Management  Skills  &  Processes 

41 

1.07 

3.43 

11 

1 

4.39 

0.6 

70 

.00000** 

TC21 

Identify  &  Protect  Promising  Technologies 

41 

1.01 

3.33 

12 

34 

2.94 

0.96 

67 

.05105  | 

TC  2 

Technology  Development  Strategy/ Acquisition  Strategy 

41 

1.00 

3.28 

13 

11 

3.96 

0.78 

70 

.00039** 

TC22 

T&E  Integration,  Strategy,  &  Planning 

41 

1.10 

3.07 

14 

8 

4.06 

0.67 

67 

.00000**| 

TC23 

Development  Test  &  Evaluation  (DT&E) 

41 

1.17 

3.04 

15 

16 

3.84 

0.75 

67 

.00023**j 

TC12 

System  Integration 

41 

1.30 

2.94 

16 

23 

3.32 

0.88 

68 

.10273  | 

TC  9 

Configuration  Management,  Data  Management,  and 
Information  Management 

41 

0.87 

2.93 

17 

19 

3.46 

0.88 

68 

.00274** 

TC15 

Systems  Engineering  Plan  (SEP) 

41 

1.03 

2.80 

18 

27 

3.19 

0.82 

67 

.04593*  | 

TC16 

Software  Developmnet,  Acquisition  Management  Technical 

Fundamentals,  Quality  &  Measurement 

41 

1.17 

2.78 

19 

20 

3.45 

0.83 

67 

.00208** 

TC31 

Business  Financial  Planning  &  Mngt;  Cost  Estimating 

41 

1.08 

2.74 

20 

5 

4.13 

0.76 

68 

.00000**| 

TC24 

Operational  Test  &  Evaluation  (OT&E)  and  Vulnerability 
Testing 

41 

1.26 

2.49 

21 

18 

3.66 

0.75 

67 

.00000** 

TC  6 

Joint/Cross  Agency/Intemational  Progrma  Magamenet  by 

U.S.  Executive  Agency 

41 

0.98 

2.43 

22 

35 

2.92 

0.89 

65 

.01099* 

TC32 

Earned  Value  Management  (EVM) 

41 

1.07 

2.40 

23 

14 

3.90 

0.88 

69 

.00000**| 

TC30 

Performance-based  Service  Agreements 

41 

1.09 

2.39 

24 

21 

3.37 

0.8 

69 

.00000**| 

TC25 

Life-Cycle  Logistic  Management,  Product  Support,  and 
Interoperabililty 

41 

1.12 

2.37 

25 

15 

3.89 

0.83 

67 

.00000** 

TCI  1 

Information  Systems:  Network  Security/ Assurance, 

Architecture,  Performance,  Infrastructure  Design,  &  System 
Mngt 

41 

1.05 

2.35 

26 

24 

3.30 

1.01 

68 

.00001** 

TC17 

Sofware  Process  Maturity,  Critical  Requirements,  Data 
Management,  Software  Support  &  Safety 

41 

1.00 

2.33 

27 

21 

3.37 

0.83 

68 

.00000** 

TC26 

Life-cycle  Cost  Optimization,  Data  Management,  & 

Integrated  Supply  Chain  Management 

41 

1.00 

2.29 

28 

17 

3.73 

0.91 

68 

.00000** 

TC27 

Logistics  Footprint  Minimization  Life-cycle  Assessment,  & 

Disposal 

41 

0.88 

2.29 

28 

30 

3.15 

0.85 

67 

.00000** 

TC34 

Industrial  Base  Assessment 

41 

0.88 

2.29 

28 

32 

3.09 

0.92 

66 

TC  4 

Life-Cycle  Cost  (Total  Ownership  Cost)  Management 

41 

0.94 

2.28 

31 

13 

3.91 

0.72 

69 

TC18 

Software  Reliability,  Reuse  and  SIS  Independent  Expert 
Reviews 

41 

1.04 

2.22 

32 

27 

3.19 

0.81 

67 

.00000** 

TC  7 

Market  Research 

41 

0.83 

2.18 

33 

36 

2.80 

0.7 

68 

.00015**1 

TC35 

Plan  Production 

41 

1.03 

2.10 

34 

26 

3.29 

0.81 

66 

.00000**1 

TC36 

Produce  Product 

41 

1.13 

1.96 

35 

24 

3.30 

0.82 

65 

.00000**1 

TC10 

Information  Resource  Strategy  and  Planning,  System 

Lifecycle,  and  Management/Technology  Awareness 

41 

0.76 

1.88 

36 

29 

3.18 

0.87 

68 

.00000** 

*  p  <  .05 


**p<  .01 

Competencies  rated  "more  critical"  by  S&T  PMs;  all  other  rated  "less  critical" 
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Descriptive  Statistics. 

The  first  observation  of  the  criticality  rankings  involves  analyzing  just  the  S&T 
PM’s  own  rankings  of  technical  competencies.  In  instances  where  the  means  are  equal, 
the  rankings  reflect  tie  scores.  For  example,  three  technical  competencies  all  share  a 
common  criticality  score  (mean  of  3.55),  and  therefore  are  all  ranked  7th ,  with  the  next 
competency  criticality  score  (mean  of  3.44)  ranked  10th.  This  ranking  methodology  is 
applied  consistently  to  both  studies  through  the  36  technical  competencies  and  27 
professional  competencies. 

The  second  observation  involves  analyzing  the  differences  of  the  technical 
competency  rankings — or  criticalities — between  the  two  sample  groups.  It  is  important 
to  note  that  although  the  differences  in  rankings  might  initially  appear  to  suggest 
significant  difference  in  the  perceived  criticality  of  certain  competencies  between  the  two 
PM  groups,  the  true  test  of  significance  must  be  statistically  computed  based  on  the 
actual  difference  in  their  means  (the  results  of  which  are  presented  and  analyzed  under 
the  Inferential  Statistics  subheading.)  For  example,  TC21 — Identify  &  Protect  Promising 
Technologies — indicates  its  criticality  ranking  by  the  S&T  PMs  as  12th  compared  with 
34th  by  the  AT&L  PMs.  However,  the  true  difference  in  the  criticality  scores  (based  on 
testing  the  difference  between  the  mean  values)  is  not  statistically  significant.  The 
criticality  rankings  must  be  analyzed  in  the  context  of  each  sample  group’s  overall  spread 
of  scores,  theoretically  ranging  from  a  minimum  value  of  1  (least  critical)  up  to  a 
maximum  value  of  5  (most  critical).  Figure  5  shows  this  difference  across  all  36 
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technical  competencies  based  on  the  means  of  the  two  sample  groups  and  their  actual 
minimum  and  maximum  values. 


Figure  5:  Technical  Competency  Response  Ranges 


In  all  but  two  cases,  the  AT&L  PMs  had  higher  criticality  scores  for  the  technical 
competencies  in  the  model,  and  their  spread  across  all  36  competencies  ranged  from  2.8 
up  to  4.39.  The  S&T  PMs  had  a  much  larger  spread  across  the  competencies,  ranging  in 
criticality  scores  from  1.88  up  to  4.34.  Table  7  also  presents  the  standard  deviations  (sd) 
for  each  of  the  competency  criticality  scores,  providing  insight  into  the  amount  by  which 
participants  within  each  study  group  vary  or  differ  in  opinion  regarding  the  criticality  of 
each  competence.  From  these  descriptive  statistics,  the  research  can  use  inferential 
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statistics  to  explore  the  investigative  research  question  of  whether  the  differences 
between  the  AT&L  PM  opinions  and  the  S&T  PM  opinions  are  statistically  significant. 

Inferential  Statistics. 

A  statistical  hypothesis  test  was  used  to  analyze  the  significance  in  the  difference 
between  the  means  of  each  competency’s  criticality  scores.  As  explained  in  the 
methodology  section  of  Chapter  3,  before  proceeding  with  a  significance  test,  four 
conditions  had  to  be  assessed  (McClave,  Benson,  Sincich,  2005:486): 

•  independence  or  dependence  of  the  samples 

•  random  or  not  random  selection  of  the  samples 

•  nonnal  or  abnormal  distribution  of  the  data 

•  equality  or  inequality  of  the  variances 

The  study  samples  were  independent  and  addressed  the  purposive  lack  of  random 
sampling,  but  the  normality  of  the  distributions  and  equality  of  the  variances  needed 
further  analysis.  Original  data  was  not  available  to  test  these  conditions  for  the  AT&L 
samples,  but  because  the  sample  sizes  are  considered  ‘large’  [(ni=65-69)  >  30]  the 
research  can  proceed  based  on  the  Central  Limit  Theorem  which  “guarantees  that  the 
sampling  distribution  of  ( X  \  -  X  2)  will  be  approximately  normal  regardless  of  the 
shapes  of  the  underlying  probability  distributions  of  the  populations  (McClave,  Benson, 
and  Sincich,  2005:483).  Although  the  S&T  sample  size  is  also  considered  ‘large’ 
[(n2=41)  >  30],  the  conservative  approach  is  to  still  test  for  normality  with  data  available. 

Testing  normality  of  the  S&T  sample  distributions  using  mathematical  analysis, 
specifically  the  Shapiro-Wilk  test,  resulted  in  /^-values  <  .05  (rejection  region)  for  32  of 
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the  36  technical  competencies,  with  20  of  those  resulting  in  /^-values  <  .01.  However,  a 
drawback  to  the  Shapiro-Wilk  test  when  sample  sizes  are  larger  is  the  “tendency  to  get 
significant  results  from  small  deviations  in  normality — thus  masking  whether  the 
deviation  from  normality  is  enough  to  bias  statistical  procedures  (Field,  2005:93). 
Therefore,  balancing  mathematical  analysis  with  graphical  analysis  is  useful.  From 
analyzing  histograms,  stem-and-leaf,  and  Q-Q  Plot  displays  of  the  distributions,  some  of 
the  frequency  distributions  suggest  slight  to  strong  skew:  16  skewing  right — or  positive, 
and  5  skewing  left — or  negative;  with  3  suggesting  possible  bi-modal  distributions. 
However,  in  most  of  these  graphical  displays,  the  Q-Q  Plots  appeared  approximately 
nonnal.  Thus,  it  is  worth  noting  the  distributions  are  not  all  consistently  normal  and 
therefore  might  benefit  from  additional  exploration  as  to  why  not;  however,  by  applying 
the  standards  of  the  Central  Limit  Theorem  based  on  the  large  sample  size,  the  data  can 
meet  the  assumption  of  approximate  nonnality  for  purposes  of  applying  the  t-test  for 
statistical  significance. 

The  final  condition  for  using  statistical  significance  tests  was  the  assumption  of 
equality  or  inequality  of  the  two  variances.  Looking  at  the  results  in  Table  7,  the 
variances  were  obviously  not  equal.  However,  the  2-tailed  F-distribution  test,  based  on 
the  ratio  of  the  two  variances,  was  necessary  to  determine  which  fonnula  to  apply  for 
degrees  of  freedom  (v)  to  use  in  combination  with  the  test-statistic  for  analyzing  the 
probability  (/;- value)  of  statistical  significance  between  each  competency’s  criticality 
scores. 
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Therefore,  once  all  four  of  the  conditions  were  addressed  for  use  of  the  t-test,  the 
/i- value  was  computed  using  a  2-tail  test  as  shown  in  Table  7,  with  30  of  the  36  having 
statistically  significant  difference  in  their  means:  5  having  statistical  significance  at  the 
.05  level,  and  25  having  statistical  significance  at  the  .01  level.  The  competencies 
without  any  statistical  difference  in  the  criticality  scores  between  the  two  sample  groups 
are  presented  in  Table  8.  Communications  Management  &  IPT/IPPT  Process  scored  the 
highest  in  both  sample  groups.  The  next  three  competencies — two  Contracting-related 
competencies  and  Technical  Process — were  within  the  top  third  tier  of  both  sample 
groups.  The  last  two — Systems  Integration  and  Identify  &  Protect  Promising 
Technologies — were  ranked  very  differently,  but  did  not  have  statistically  significant 
differences  between  their  mean  criticality  scores. 


Table  8:  Technical  Competencies  without  Statistical  Significance 


TECHNICAL 

r'OA/TPT7MTT7Mr'V  TTTT  Th  Q 

S&T  Results 

AT&L  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

TC  8 

Communications  Management  & 
IPT/IPPT  Process 

41 

0.79 

4.34 

1 

1 

4.39 

0.71 

69 

.74814 

TC28 

Contract  Approach,  Requirements  & 
Supporting  Documents,  Prepare  & 
Issue  Solicitation 

41 

1.01 

3.80 

3 

7 

4.08 

0.75 

68 

.13427 

TC29 

Source  Selection,  Contract  Award, 
Contract  Administration,  &  Contract 
Closeout 

41 

0.84 

3.73 

5 

9 

3.99 

0.79 

69 

.11626 

TC14 

Technical  Process 

41 

1.03 

3.61 

6 

12 

3.95 

0.71 

68 

.06772 

TC21 

Identify  &  Protect  Promising 
Technologies 

41 

1.01 

3.33 

12 

34 

2.94 

0.96 

67 

.05105 

TC12 

System  Integration 

41 

1.30 

2.94 

16 

23 

3.32 

0.88 

68 

.10273 

The  next  set  of  technical  competencies  (five  total)  did  have  statistically  significant 
differences  at  the  .05  level  between  their  mean  criticality  scores,  as  shown  in  Table  9: 
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Technical  Competencies  with  Statistical  Significance  at  the  .05  Level.  One  of  the 
technical  competencies  in  this  set — S&T  Goal,  Program  Considerations,  &  Technology 
Engineering — was  rated  “more  critical”  by  S&T  program  managers  than  traditional 
acquisition  program  managers.  The  five  others  were  rated  “less  critical”  by  the  S&T 
program  managers,  the  most  surprising  of  which  was  Technical  Management  Process. 
However,  across  the  total  ratings  by  the  S&T  group  it  ranked  4th  out  of  36,  further 
demonstrating  how  the  S&T  group  generally  gave  lower  ratings  across  the  whole  set  of 
competencies. 


Table  9:  Technical  Competencies  with  Statistical  Significance  at  the  .05  Level 


TECHNICAL  COMPENTENCY 

S&T  Results 

AT&L  Results 

Level  of 
Signif. 

ill  LLO 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

TCI  3 

Technical  Management  Process 

41 

0.86 

3.79 

4 

5 

4.13 

0.6 

68 

.03305* 

TCI  9 

S&T  Goal,  Program  Considerations,  & 
Technology  Engineering 

41 

1.08 

3.55 

7 

31 

3.13 

0.9 

68 

.03908* 

TC33 

Financial  Reporting  &  Oversight, 

Dpt/ Agency  Programming,  Planning,  & 
Budgeting  System 

41 

1.04 

3.55 

7 

9 

3.99 

0.8 

68 

.02202* 

TCI  5 

Systems  Engineering  Plan  (SEP) 

41 

1.03 

2.80 

18 

27 

3.19 

0.8 

67 

.04593* 

TC  6 

Joint/Cross  Agency/Intemational 
Progrma  Magamenet  by  U.S.  Executive 
Agency 

41 

0.98 

2.43 

22 

35 

2.92 

0.9 

65 

.01099* 

*  p  <  .05 


Competencies  rated  "more  critical"  by  S&T  PMs;  all  other  rated  "less  critical" 


The  differences  in  criticality  scores  for  the  remaining  25  competencies  were 
statistically  significant  at  the  .01  level,  all  rated  by  S&T  PMs  as  “less  critical”  except  for 
Transition  Techniques  which  they  rated  as  “more  critical”  and  ranked  2nd  (see  Table  10). 
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Table  10:  Technical  Competencies  with  Statistical  Significance  at  the  .01  Level 


TECHNICAL  COMPENTENCY  TITLES 

S&T  Results 

AT&L  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

TC20 

Transition  Techniques 

41 

0.83 

3.88 

2 

33 

3.04 

0.94 

68 

.00000**  | 

TC  1 

Requirements  Process 

41 

0.88 

3.55 

7 

3 

4.28 

0.64 

68 

.00002**  j 

TC  5 

Risk  Management 

41 

0.86 

3.44 

10 

4 

4.17 

0.75 

69 

.00002**  j 

TC  3 

Core  Management  Skills  &  Processes 

41 

1.07 

3.43 

11 

1 

4.39 

0.6 

70 

.00000**  j 

TC  2 

Technology  Development  Strategy/ Acquisition 
Strategy 

41 

1.00 

3.28 

13 

11 

3.96 

0.78 

70 

.00039** 

TC22 

T&E  Integration,  Strategy,  &  Planning 

41 

1.10 

3.07 

14 

8 

4.06 

0.67 

67 

.00000**  j 

TC23 

Development  Test  &  Evaluation  (DT&E) 

41 

1.17 

3.04 

15 

16 

3.84 

0.75 

67 

.00023**  j 

TC9 

Configuration  Management,  Data  Management,  and 
Information  Management 

41 

0.87 

2.93 

17 

19 

3.46 

0.88 

68 

.00274** 

TC16 

Software  Developmnet,  Acquisition  Management 
Technical  Fundamentals,  Quality  &  Measurement 

41 

1.17 

2.78 

19 

20 

3.45 

0.83 

67 

.00208** 

TC31 

Business  Financial  Planning  &  Mngt;  Cost 
Estimating 

41 

1.08 

2.74 

20 

5 

4.13 

0.76 

68 

.00000** 

TC24 

Operational  Test  &  Evaluation  (OT&E)  and 
Vulnerability  Testing 

41 

1.26 

2.49 

21 

18 

3.66 

0.75 

67 

.00000** 

TC32 

Earned  Value  Management  (EVM) 

41 

1.07 

2.40 

23 

14 

3.90 

0.88 

69 

.00000**  j 

TC30 

Performance-based  Service  Agreements 

41 

1.09 

2.39 

24 

21 

3.37 

0.8 

69 

.00000**  j 

TC25 

Life-Cycle  Logistic  Management,  Product  Support, 
and  Interoperabililty 

41 

1.12 

2.37 

25 

15 

3.89 

0.83 

67 

.00000** 

TCI  1 

Information  Systems:  Network  Security/Assurance, 
Architecture,  Performance,  Infrastructure  Design,  & 
System  Mngt 

41 

1.05 

2.35 

26 

24 

3.30 

1.01 

68 

.00001** 

TCI  7 

Sofware  Process  Maturity,  Critical  Requirements, 
Data  Management,  Software  Support  &  Safety 

41 

1.00 

2.33 

27 

21 

3.37 

0.83 

68 

.00000** 

TC26 

Life-cycle  Cost  Optimization,  Data  Management,  & 
Integrated  Supply  Chain  Management 

41 

1.00 

2.29 

28 

17 

3.73 

0.91 

68 

.00000** 

TC27 

Logistics  Footprint  Minimization  Life-cycle 
Assessment,  &  Disposal 

41 

0.88 

2.29 

28 

30 

3.15 

0.85 

67 

.00000** 

TC34 

Industrial  Base  Assessment 

41 

0.88 

2.29 

28 

32 

3.09 

0.92 

66 

.00002**  j 

TC  4 

Life-Cycle  Cost  (Total  Ownership  Cost) 
Management 

41 

0.94 

2.28 

31 

13 

3.91 

0.72 

69 

.00000** 

TC18 

Software  Reliability,  Reuse  and  SIS  Independent 
Expert  Reviews 

41 

1.04 

2.22 

32 

27 

3.19 

0.81 

67 

.00000** 

TC  7 

Market  Research 

41 

0.83 

2.18 

33 

36 

2.80 

0.7 

68 

.00015**1 

TC35 

Plan  Production 

41 

1.03 

2.10 

34 

26 

3.29 

0.81 

66 

.00000**  j 

TC36 

Produce  Product 

41 

1.13 

1.96 

35 

24 

3.30 

0.82 

65 

.00000**  j 

TC10 

Information  Resource  Strategy  and  Planning, 

System  Lifecycle,  and  Management/Technology 
Awareness 

41 

0.76 

1.88 

36 

29 

3.18 

0.87 

68 

.00000** 

**p<  .01 

Competencies  rated  "more  critical"  by  S&T  PMs;  all  other  rated 

less  critical" 
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Overall  Analysis. 

The  results  and  analysis  show  that  the  perceived  criticality  of  certain  program 
management  competencies  differs — at  least  between  the  traditional  acquisition  PMs  from 
the  AT&L  study  and  the  S&T  PMs  interviewed  in  this  study.  Of  the  36  technical 
competencies  from  the  AT&L  study,  30  had  statistically  significant  differences  between 
the  criticality  ratings:  S&T  PMs  rated  two  of  those  as  “more  critical”  and  the  other  28  as 
“less  critical.”  The  two  “more  critical”  competencies — Transition  Techniques  and  S&T 
Goal,  Program  Considerations,  &  Technology  Engineering — ranked  2nd  and  7th 
respectively  for  S&T  PMs;  however,  those  happened  to  be  two  of  the  “least  critical”  and 
lowest  ranked  in  the  AT&L  study — 33rd  and  3 1st  respectively.  Perhaps  even  more 
noteworthy  than  the  individual  scores,  was  the  much  lower  spread  of  rankings  for  the  28 
“less  critical”  competencies  by  the  S&T  PMs. 

Two  important  observations  emerged  through  the  interview  process  potentially 
related  to  the  28  low-rated  competencies.  The  first  observation  pertained  to  the 
respondents’  selection  of  importance  and  frequency  ratings:  in  several  cases,  respondents 
struggled  with  their  selection  based  on  lack  of  total  agreement  with  the  definitions  and 
behaviors  described  under  each  competency  heading.  For  example,  references  to 
“acquisition  strategy,”  “milestone  approval  authority,”  or  “earned  value  management” 
often  elicited  responses  such  as  “we  don’t  do  that”  or  “that’s  only  in  the  product  centers.” 
Thus,  most  respondents  would  automatically  rate  the  competency  much  lower  in 
importance  and  with  either  a  “never”  or  “sometimes”  frequency  rating.  In  some  cases, 
however,  the  respondents  indicated  they  recognized  that  although  the  exact  wording  of 
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the  descriptions  consisted  of  more  traditional  acquisition  terminology,  the  theory  or  intent 
was  important  and  applied  within  S&T  program  management,  albeit  “by  another  name.” 
Thus,  their  ratings  for  importance  and  frequency  shifted  upward. 

The  second  observation  that  emerged  through  the  SME  interview  process  was  the 
struggle  that  several  of  the  S&T  PMs  seemed  to  have  with  the  second  question  pertaining 
to  frequency:  “How  often  is  this  competency  used  in  performing  work?”  Several  of  the 
interviewees  sought  clarification  whether  their  responses  should  reflect  “ideally”  or  “in 
reality.”  This  implied  disconnect  was  more  directly  stated  by  respondents  who  made 
comments  such  as  “we  really  should  be  doing  this  more  often  in  S&T  but  we  don’t.” 
Comments  such  as  this  suggest  an  opportunity  for  examination  or  clarification  with 
organizational  leadership  regarding  the  linkage  between  strategic  vision  and  S&T 
program  management  perfonnance  expectations. 

It  is  not  known  precisely  how  much  influence  the  perceived  ambiguities,  alternate 
perspectives,  or  individual  uncertainties  could  have  shaped  all  the  SME’s  ratings, 
especially  without  interviewer  insight  to  the  SMEs  responding  through  asynchronous 
online  interviews.  However,  the  insights  that  were  gained  could  possibly  help  explain  the 
consistently  lower  28  criticality  ratings  and/or  the  16  positively  skewed  distributions. 
Lower  importance  and  frequency  ratings  are  not  problems  by  themselves;  the  important 
question  that  needs  addressing  is  “why?”  If  indeed  certain  competencies  are  less  critical 
to  S&T  program  managers,  then  that  is  very  useful  data  to  know  when  developing  a 
competency  model  by  which  workforce  management  initiatives  are  shaped.  However,  if 
competencies  are  ranked  lower  due  to  construct  validity  issues — either  by  the  definitions 


58 


themselves,  ambiguity  in  the  phrasing  of  the  questions,  or  a  disconnect  in  the  “reality”  of 
competency  expectations  versus  application — then  the  reliability  of  the  results  becomes 
questionable. 

Part  III:  Professional  Competencies 

The  third  part  of  this  research  study  examines  perceptions  about  the  criticality  of 
professional  competencies  to  successful  program  management.  The  data  collected  were 
based  on  the  same  set  of  questions  as  the  technical  competencies — 5-point  Likert  scales 
about  the  importance  and  frequency  for  each  competency.  The  average  of  these  two 
means  computed  the  overall  criticality  score  (mean)  as  shown  and  ranked  in  Table  11. 
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Table  11:  Comparison  of  Professional  Competencies 


PROFESSIONAL 

S&T  PM  Results 

AT&L  PM  Results 

Level  of 
Signif. 

V^/wlVlr  J_/i>  1  1_/1>V^  I  111EDO 

N 

SD 

Mean 

Rank 

Rank 

Mean 

SD 

N 

(p  -value) 

PC16 

Oral  Communication 

41 

0.59 

4.48 

1 

1 

4.51 

0.90 

69 

.80955 

PC27 

Written  Communication 

41 

0.81 

4.38 

2 

17 

4.10 

1.09 

69 

.13074 

PC  1 

Accountability 

41 

0.70 

4.37 

3 

10 

4.30 

1.02 

69 

.68927 

PC19 

Problem  Solving 

41 

0.64 

4.33 

4 

9 

4.39 

0.79 

69 

.66037 

PCM 

Interpersonal  Skills 

41 

0.73 

4.27 

5 

3 

4.45 

0.99 

69 

.27442 

PC23 

Team  Building 

41 

0.76 

4.09 

6 

2 

4.46 

0.78 

69 

.01507* 

PC  5 

Customer  Service 

41 

0.91 

4.02 

7 

15 

4.14 

0.97 

69 

.53258 

PC10 

Financial  Management 

41 

0.77 

3.98 

8 

20 

3.46 

1.38 

69 

.01359* 

PC24 

Technical  Credibility 

41 

0.83 

3.93 

9 

14 

4.16 

0.98 

69 

.18757 

PC13 

Influencing  &  Negotiating 

41 

0.87 

3.91 

10 

3 

4.45 

0.74 

69 

.00145**| 

PC  6 

Decisiveness 

41 

0.78 

3.90 

11 

6 

4.43 

0.96 

69 

.00217**| 

PC11 

Flexibility 

41 

0.85 

3.90 

11 

3 

4.45 

0.92 

69 

.00205**| 

PC17 

Partnering 

41 

0.85 

3.88 

13 

6 

4.43 

0.74 

69 

.00091**| 

PC  4 

Creativity  &  Innovation 

41 

0.87 

3.87 

14 

16 

4.12 

1.16 

69 

,19590  | 

PC21 

Resilience 

41 

0.80 

3.79 

15 

8 

4.41 

0.81 

69 

PC  7 

Developing  Others 

41 

0.95 

3.74 

16 

21 

3.41 

1.34 

69 

^B05SM 

PC22 

Strategic  Thinking 

41 

0.92 

3.72 

17 

11 

4.25 

0.96 

69 

.00513**| 

PC25 

Technology  Mngt 

41 

0.98 

3.61 

18 

27 

3.03 

1.31 

69 

.00979**1 

PC  3 

Continual  Learning 

41 

0.84 

3.59 

19 

22 

3.35 

1.37 

69 

.26567  | 

PC  2 

Conflict  Management 

41 

0.91 

3.57 

20 

17 

4.10 

1.09 

69 

.00775** 

PC12 

Human  Capital  Mngt 

41 

1.12 

3.57 

20 

25 

3.12 

1.44 

69 

.06885  | 

PC26 

Vision 

41 

1.05 

3.50 

22 

11 

4.25 

0.98 

69 

.00039** 

PC  9 

External  Awareness 

41 

0.94 

3.41 

23 

19 

3.84 

1.38 

69 

.05791 

PC  8 

Entrepreneurship 

41 

0.98 

3.38 

24 

23 

3.32 

1.22 

69 

.78489 

PC18 

Political  Savvy 

41 

0.98 

3.37 

25 

13 

4.23 

1.00 

69 

.00003** 

PC15 

Leveraging  Diversity 

41 

1.02 

3.28 

26 

24 

3.22 

1.32 

69 

.78912 

PC20 

Public  Service  Motivation 

41 

1.09 

2.73 

27 

26 

3.07 

1.31 

69 

.14821 

*  p  <  .05 


Competencies  rated  "more  critical”  by  S&T  PMs;  all  other  rated  "less  critical" 


Descriptive  Statistics. 

The  resulting  rankings  of  the  mean  criticality  scores  according  to  SMEs  from 
S&T  program  management  are  compared  in  Table  1 1  with  those  from  the  AT&L  study 
group  comprised  of  predominantly  traditional  acquisition  program  managers.  The  same 
underlying  assumption  applies  regarding  the  prioritized  rankings:  the  highest  professional 
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competency  presumably  has  the  greatest  affect  on  PM  performance,  theoretically 
appearing  in  an  empirical  study  as  having  the  largest  regression  coefficient  ([]). 

The  first  observation  about  the  descriptive  statistics  examines  the  means  of  the 
S&T  PM  criticality  scores  in  relation  to  themselves.  Again,  in  instances  where  the  means 
are  equal,  the  rankings  reflect  tie  scores.  The  second  observation  compares  the  difference 
in  the  rank  ordering  of  the  means  between  the  two  groups.  It  should  be  reemphasized 
that  the  difference  in  the  rankings  of  the  means  alone  cannot  detennine  statistical 
significance  (such  inferential  statistical  analysis  follows  under  the  next  subheading.)  For 
example,  the  2nd  highest  ranked  professional  competency  according  to  S&T  PMs  is 
“Written  Communication.”  Despite  its  rank  order  only  being  17th  according  to  AT&L 
respondents,  this  difference  is  not  statistically  significant  when  the  value  of  the  two 
means — 4.28  and  4.10,  respectively — are  tested. 

Like  the  technical  competencies,  the  research  can  benefit  from  additional  insights 
to  be  gained  by  graphically  analyzing  the  difference  in  the  range  of  responses  between 
the  two  sample  groups,  as  shown  in  Figure  6.  Unlike  the  stark  differences  that  exist 
regarding  the  technical  competencies,  the  professional  competencies  tend  to  align  closer 
between  the  two  sample  groups.  The  next  comparison  to  make  between  the  sample 
groups  is  whether  any  differences  between  these  mean  scores  are  statistically  significant, 
as  determined  through  inferential  statistics. 
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Figure  6:  Professional  Competency  Response  Ranges 
Inferential  Statistics. 

The  same  statistical  hypothesis  test  applied  for  the  technical  competencies  was 
also  used  to  test  the  professional  competencies.  In  order  to  use  the  more  conservative  t- 
test,  the  four  assumptions — independence,  randomness,  normality,  and  equality  of 
variances — were  addressed  through  the  same  analysis  processes  explained  in  Part  II. 
With  the  proper  degrees  of  freedom  detennined  for  use  with  the  computed  test  statistic, 
the  resulting  probability  (p-value)  of  statistical  significance  could  be  analyzed. 

Of  the  27  professional  competencies,  15  had  no  statistical  significance  in  the  different 
ratings  by  the  two  sample  groups,  as  shown  in  Table  12. 


62 


Table  12:  Professional  Competencies  without  Statistical  Significance 


PROFESSIONAL 

COMPENTENCY  TITLES 

S&T  PM  Results 

AT&L  PM  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

PC16 

Oral  Communication 

41 

0.59 

4.48 

1 

1 

4.51 

0.90 

69 

.80955 

PC27 

Written  Communication 

41 

0.81 

4.38 

2 

17 

4.10 

1.09 

69 

.13074 

PC  1 

Accountability 

41 

0.70 

4.37 

3 

10 

4.30 

1.02 

69 

.68927 

PC19 

Problem  Solving 

41 

0.64 

4.33 

4 

9 

4.39 

0.79 

69 

.66037 

PCM 

Interpersonal  Skills 

41 

0.73 

4.27 

5 

3 

4.45 

0.99 

69 

.27442 

PC  5 

Customer  Service 

41 

0.91 

4.02 

7 

15 

4.14 

0.97 

69 

.53258 

PC24 

Technical  Credibility 

41 

0.83 

3.93 

9 

14 

4.16 

0.98 

69 

.18757 

PC  4 

Creativity  &  Innovation 

41 

0.87 

3.87 

14 

16 

4.12 

1.16 

69 

.19590 

PC  7 

Developing  Others 

41 

0.95 

3.74 

16 

21 

3.41 

1.34 

69 

.13058 

PC  3 

Continual  Learning 

41 

0.84 

3.59 

19 

22 

3.35 

1.37 

69 

.26567 

PC12 

Human  Capital  Mngt 

41 

1.12 

3.57 

20 

25 

3.12 

1.44 

69 

.06885 

PC  9 

External  Awareness 

41 

0.94 

3.41 

23 

19 

3.84 

1.38 

69 

.05791 

PC  8 

Entrepreneurship 

41 

0.98 

3.38 

24 

23 

3.32 

1.22 

69 

.78489 

PC15 

Leveraging  Diversity 

41 

1.02 

3.28 

26 

24 

3.22 

1.32 

69 

.78912 

PC20 

Public  Service  Motivation 

41 

1.09 

2.73 

27 

26 

3.07 

1.31 

69 

.14821 

The  remaining  12  professional  competencies  did  have  statistical  significance  in 
the  differences  between  their  means,  or  criticality  ratings,  with  2  having  statistical 
significance  at  the  .05  level:  Team  Building  which  was  rated  “less  critical”  by  the  S&T 
PMs,  and  Financial  Management  which  was  rated  “more  critical”  by  the  S&T  PMs,  as 
shown  in  Table  13.  The  increase  in  the  Financial  Management  professional  competency 
might  be  surprising  to  some  researchers,  considering  the  much  higher-dollar  program 
costs  and  high-visibility  budget  reviews  required  in  traditional  acquisition  program 
management.  However,  one  hypothesis  for  this  result  is  that  traditional  acquisition  PMs 
have  strict  oversight  and  guidelines  limiting  their  actual  control  over  funding  decisions, 
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whereas  S&T  PMs  have  much  broader  latitude  and  thus  individual  accountability  for 
their  investment  portfolio  decisions. 


Table  13:  Professional  Competencies  with  Statistical  Significance  at  the  .05  Level 


PROFESSIONAL 

COMPENTENCY  TITLES 

S&T  PM  Results 

AT&L  PM  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

PC23 

Team  Building 

41 

0.76 

4.09 

6 

2 

4.46 

0.78 

69 

.01507* 

PC10 

Financial  Management 

41 

0.77 

3.98 

8 

20 

3.46 

1.38 

69 

.01359* 

*  p  <  .05 


Competencies  rated  "more  critical"  by  S&T  PMs;  all  other  rated  "less  critical" 


The  remaining  10  professional  competencies  were  found  to  be  significantly 
different  at  the  .01  level  between  S&T  PMS  and  the  AT&L  PMs,  as  shown  in  Table  14. 
Technology  Management  was  the  only  professional  competency  rated  as  “more  critical” 
by  S&T  PMs,  with  the  other  nine  rated  as  “less  critical”  by  S&T  PMs. 


Table  14:  Professional  Competencies  with  Statistical  Significance  at  the  .01  Level 


PROFESSIONAL 

COMPENTENCY  TITLES 

S&T  PM  Results 

AT&L  PM  Results 

Level  of 
Signif. 

n 

sd 

mean 

rank 

rank 

mean 

sd 

n 

( p  -value) 

PC  13 

Influencing  &  Negotiating 

41 

0.87 

3.91 

10 

3 

4.45 

0.74 

69 

.00145**  | 

PC  6 

Decisiveness 

41 

0.78 

3.90 

11 

6 

4.43 

0.96 

69 

.00217**  1 

PC11 

Flexibility 

41 

0.85 

3.90 

11 

3 

4.45 

0.92 

69 

.00205**  | 

PC  17 

Partnering 

41 

0.85 

3.88 

13 

6 

4.43 

0.74 

69 

.00091**  | 

PC21 

Resilience 

41 

0.80 

3.79 

15 

8 

4.41 

0.81 

69 

.00019**  1 

PC22 

Strategic  Thinking 

41 

0.92 

3.72 

17 

11 

4.25 

0.96 

69 

.00513**  1 

PC25 

Technology  Mngt 

41 

0.98 

3.61 

18 

27 

3.03 

1.31 

69 

.00979**  | 

PC  2 

Conflict  Management 

41 

0.91 

3.57 

20 

17 

4.10 

1.09 

69 

.00775**  | 

PC26 

Vision 

41 

1.05 

3.50 

22 

11 

4.25 

0.98 

69 

.00039**  1 

PC  18 

Political  Savvy 

41 

0.98 

3.37 

25 

13 

4.23 

1.00 

69 

.00003**  | 

Competencies  rated  "more  critical"  by  S&T  PMs;  all  other  rated  "less  critical" 
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Part  IV:  Open-Ended  Questions 

The  final  part  of  the  study  involves  a  qualitative  analysis  of  the  responses  to  the 
three  open-ended  questions  based  on  a  grounded  theory  approach.  Grounded  theory  is  an 
inductive  method  of  analysis,  leading  to  the  emergence  of  theories  through  consideration 
and  analysis  of  the  data  (Patten,  2005: 153).  For  this  research,  the  goal  of  the  three  open- 
ended  questions  was  to  identify  themes  or  issues  which  might  further  elucidate  the  S&T 
program  management  environment  regarding  the  applications  of  competency  modeling 
and  its  potential  impact  on  future  human  resource  management  decisions.  The  results 
and  analysis  of  the  data  are  segmented  by  each  open-ended  question  below. 

Question  1. 

The  first  open-ended  question  asked  the  respondents  whether  any  competencies 
were  missing  from  the  current  AT&L  PM  model  that  they  considered  to  be  critical  to 
managing  S&T  programs.  Of  the  41  respondents,  24  (56%)  indicated  there  were  no 
additional  competencies  needing  to  be  added;  however,  17  (42%)  made  suggestions  about 
elements  they  thought  were  missing  from  the  model.  The  suggestions  as  originally 
worded  by  the  respondents  are  provided  in  Appendix  C:  Comments  to  Open-Ended 
Question  #1,  with  the  attempt  to  group  them  by  technical  competencies  and  professional 
competencies. 

Some  of  the  respondents  suggested  additions  that  are  already  included  in  the 
AT&L  competency  model — either  explicitly  by  the  same  name  or  implicitly  by  another 
name.  For  example,  the  suggestion  to  include  a  technical  competency  related  to  the 
development  and  use  of  architectures  is  already  specifically  addressed  within  four  of  the 
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technical  competency  definitions  (TCI,  TCI  1,  TC14,  and  TC16).  Redundant 
recommendations  like  this  could  be  attributed  to  oversight  or  memory  lapse — especially 
during  an  on-the-spot  30-minute  interview.  However,  redundant  recommendations 
should  not  be  dismissed;  rather,  their  inclusion  amplifies  the  perceived  importance  or 
relevance  of  that  particular  competency.  Such  instances  become  important  when  fine- 
tuning  a  competency  model  for  a  specific  target  group. 

Several  of  the  other  recommendations  emphasize  particular  aspects  of 
competencies  that  might  be  more  important  to  S&T  PMs.  Whether  analysis  techniques, 
process  controls,  tech  reports,  or  designs — the  emphasis  seems  to  be  on  demonstrating 
competencies  in  S&T-unique  contexts.  The  tenns  “knowledge  of’  and  “understanding 
of’  specific  subject  areas  were  often  mentioned.  An  important  progression  in 
incorporating  such  suggestions  into  an  S&T  PM  model  would  be  to  translate  them  into 
actual  behaviors  or  tasks  required  by  the  PM  which  outwardly  demonstrate  the  intrinsic 
knowledge  or  understanding.  In  other  words,  what  decisions  or  actions  must  an  S&T 
program  manager  make  that  is  critically  dependent  on  a  specific  knowledge  or 
understanding?  Questions  such  as  this  are  at  the  core  of  a  “critical  incident  technique” 
interview,  in  which  SMEs  help  define  core  competencies  in  terms  of  specific  behaviors 
resulting  in  superior  performance.  Thus,  initial  results  such  as  this  underscore  the 
importance  of  one  respondent’s  overarching  comment:  “the  formal  definitions  in  the 
model  are  not  right  for  S&T  program  managers;  the  concepts  behind  the  competencies 
are  okay”  but  they  need  specific  refinement  in  the  context  of  the  S&T  environment. 
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Question  2. 

The  second  open-ended  question  asked  of  the  respondents  was  whether  they 
thought  demonstrated  proficiency  in  a  set  of  S&T  program  manager  competencies  could 
serve  as  hiring  and/or  evaluation  criteria.  The  intent  behind  this  question  was  to  extract 
opinions  which  might  provide  insights  into  the  current  environment  affecting  future 
implementation  challenges.  Currently,  the  application  of  competency  modeling — at  least 
within  government — is  primarily  focused  on  workforce  development  initiatives.  The 
three  other  uses  for  integrating  competency  models  into  workforce  management — 
selection  (hiring),  performance  appraisal,  and  succession  planning — are  just  now  being 
tested  in  pilot  studies  like  the  examples  mentioned  in  Chapter  II. 

Of  the  41  respondents,  37  (90%)  expressed  overall  support  towards  a  specialized 
set  of  competencies  for  S&T  program  management;  however,  their  responses  often 
contained  qualifiers  or  caveats.  Four  respondents  (10%)  were  definitely  opposed  to  the 
idea.  The  individual  responses  are  listed  in  Appendix  D:  Comments  to  Open-Ended 
Question  #2,  with  an  attempt  to  sort  them  by  supportive  or  not-supportive  categories: 

Themes  that  emerged  from  the  supportive  category  include: 

•  useful  if  tailored  to  the  specific  job: 

o  by  levels  (entry,  journeyman,  advanced) 
o  by  technology  programs  (basic,  applied,  or  advanced) 

•  useful  as  an  augmentation — not  an  absolute  or  stand-alone 

o  to  help  guide  the  process  for  better  matching 
o  to  aid  the  decision-maker’s  judgment — not  replace  it 

Themes  that  emerged  from  the  not-supportive  category  include: 

•  difficulty  in  measuring  /  evaluating  competencies 

•  lack  of  standardized  or  consistent  application 


67 


An  overwhelming  majority  recognized  value  in  having  a  well-defined  and  specialized  set 
of  competencies  unique  to  their  profession.  The  greatest  uncertainties  remain  in  the 
challenge  of  quantifying  and  standardizing  them  consistently  across  the  workforce. 

Question  3. 

The  third  open-ended  question  asked  the  respondents  whether  a  technical  degree 
(science,  engineering,  mathematics)  is  critical  to  proficient  S&T  management.  The  intent 
of  this  question  was  to  assess  opinions — based  on  past  precedent  and  current  culture — 
regarding  the  necessity  and  expectation  for  S&T  managers  to  have  technical  degrees. 

This  topic  generated  a  lot  of  discussion,  often  with  the  same  respondent 
describing  examples  of  successes  and  failures  from  both  scenarios.  However,  when 
asked  for  a  definitive  “yes”  or  “no”  response,  26  (65%)  of  the  40  SMEs  (one  did  not 
commit  to  an  answer)  maintained  that  having  a  degree  was  important  enough  to  the 
success  of  an  S&T  program  manager  to  deem  it  a  “critical”  requirement.  The  general 
themes  of  concern  from  this  group  of  respondents  included:  lack  of  understanding  and 
insight  to  technical  implications  affecting  programmatic  decisions;  over-reliance  on 
contractor  technical  expertise;  and  lack  of  credibility — and  thus  impact — with  the 
technical  people  being  managed.  However,  14  (35%)  suggested  it  was  not  critical, 
although  most  agreed  it  would  definitely  be  a  benefit  to  the  PM.  Themes  expressed  from 
this  group  of  respondents  include:  technical  understanding  could  be  gained  through 
experience — not  just  a  degree;  the  business  manager  with  critical  thinking  and  analysis 
skills  could  excel  in  S&T  program  management;  a  balanced  team  of  expertise — the 
program  manager  with  an  engineering  tech  advisor — is  a  successful  model.  The  specific 
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responses  around  this  topic  are  presented  in  Appendix  E:  Comments  to  Open-Ended 
Question  #3. 

Currently  a  technical  degree  is  a  hiring  and  certification  restriction — certified 
program  managers  cannot  cross  over  to  S&T  program  management  without  a  technical 
degree  (although  military  PMs  can  get  assigned  with  or  without  technical  degrees.)  Thus, 
the  development  model  is  dependent  on  first  recruiting  scientists  and  engineers  (S&Es), 
then  training  them  to  become  program  managers.  These  current  practices  align  with 
documented  success  stories  and  related  studies,  such  as  the  1997  report  on  “The 
Perceived  Importance  of  Technical  Competence  to  Project  Managers  in  the  Defense 
Acquisition  Community”  which  concluded  that  (technology-based)  technical  competence 
was  “extremely  important  or  absolutely  essential”  especially  for  projects  in  earlier 
acquisition  phases  “demonstrating  and  validating  technology”  (Grant,  Baumgardner,  and 
Shane,  1997:17).  The  long-term  strategic  concern  over  this  paradigm  centers  on  the 
projected  diminishing  pool  of  S&E  candidates.  Can  the  DoD  afford  to  compete  in  the 
future  for  high-demand,  limited-supply  S&Es  to  manage  its  programs,  or  could  /  should 
the  paradigm  shift  to  hire  from  more  abundant  management  pools  to  then  train  in 
technology-focused  competencies? 
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V.  Conclusions  &  Recommendations 


The  purpose  of  this  exploratory  research  was  to  analyze  competencies  of  program 
managers  across  the  Department  of  Defense  (DoD)  Acquisition,  Technology,  and 
Logistics  (AT&L)  spectrum,  specifically  those  that  might  be  unique  to  Science  and 
Technology  (S&T)  program  management.  Ongoing  emphasis  on  the  importance  of 
developing  the  workforce  for  successful  performance  through  competency-based 
initiatives  is  the  number  one  goal  in  AT&L’s  2007  Strategic  Goals  Implementation  Plan. 
Thus,  an  overarching  question  of  this  research  was  whether  program  management 
competencies  differ  across  AT&L,  and  if  so,  in  which  areas?  To  scope  the  research 
effort,  an  AT&L  interim  competency  model  became  the  measure  by  which  to  compare 
competencies  for  program  managers  (PMs)  in  traditional  acquisition  programs  with  PMs 
in  S&T  programs.  The  investigative  questions  guiding  the  research  methodology  were 
whether  certain  competencies  were  more  or  less  critical  for  different  types  of  program 
manager  positions,  and  if  so,  were  the  differences  statistically  significant  enough  to 
warrant  additional  resources  towards  a  specialized  S&T  PM  competency  model. 

The  results  of  the  data  collection  and  analysis  using  the  AT&L  interim  PM 
competency  model  clearly  show  subject  matter  experts  from  traditional  acquisition 
program  management  and  those  from  S&T  program  management  do  rate  the  criticality  of 
certain  competencies  differently.  In  42  out  of  63  instances  (67%)  the  criticality  scores  of 
S&T  PMs  had  statistically  significant  differences.  Only  four  of  those  42  competencies 
were  rated  “more  critical”  by  S&T  PMs:  Transition  Techniques;  S&T  Goal,  Program 
Considerations,  &  Technology  Engineering;  Financial  Management;  and  Technology 
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Management.  The  other  38  of  the  42  (90%),  were  rated  “less  critical”  with  statistically 
significant  lower  scores  than  those  of  their  acquisition  PM  counterparts.  The  qualitative 
analysis  of  the  open-ended  interview  questions  suggest  that  although  the  AT&L  PM 
competency  model  may  seem  to  have  face  validity,  the  definitions  currently  defining  the 
competencies  may  not  have  adequate  content  validity  for  S&T  program  management.  In 
other  words,  the  competencies  defined  with  the  current  behavior-based  tasks  do  not 
represent  the  S&T  program  management  context  to  elicit  reliable  results  when  applying 
the  competency  model  to  shape  successful  performance.  Although  there  are  inherent 
sampling  biases  within  this  study  which  limit  the  generalization  of  these  results  to  the 
population  at  large,  the  results  suggest  that  enough  statistically  significant  differences 
exist  to  warrant  further  research  into  a  specialized  S&T  PM  competency  model. 

Whether  under  the  direction  of  current  federal  mandates  or  by  motivation  to  reap 
the  best  return  on  human  capital  investment,  the  DoD  cannot  afford  not  to  ensure  the 
accuracy  and  reliability  of  the  competency  models  by  which  it  will  recruit,  develop,  and 
retain  its  workforce.  The  process  for  developing  the  competency  models  aligns  with  best 
practices  across  literature;  applying  the  resources  to  ensure  the  validity  of  individual 
competencies  within  those  frameworks  will  rely  on  the  commitment  of  each  AT&L 
functional  leadership.  According  to  Michael  Ayers,  3M  workforce  management 
consultant  and  CEO  of  The  Commonwealth  Practice,  “getting  a  firm  grip  on 
competencies  will  permit  your  organization  to  have  richer  conversations  about  the 
demands  it  is  facing  in  a  changing  environment. .  .but  doing  it  effectively  requires  the 
courage  to  look  at  the  current  situation  honestly  and  the  courage  to  own  the  responsibility 
for  creating  the  future  that  the  organization  wants”  (Ayers,  2001 :2). 
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Appendix  A:  Structured  Interview 


Analysis  of  Competencies  for  Managing  Science  &  Technology  (S&T)  Programs 


Introduction 

Thank  you  in  advance  for  volunteering  to  share  your  expertise  based  on  your  personal  experience  of 
having  managed  DoD  S&T  (aka  R&D)  programs,  or  having  supervised  STMs  who  do.  Participation 
criteria:  recognized  by  your  organization  as  a  “superior-performer”  subject  matter  expert  and  have  at  least 
two  years  of  experience.  No  personally  identifiable  data  will  be  collected,  the  nature  of  die  research  is 
uon-controversial,  and  your  inputs  will  only  be  disclosed  in  a  consolidated  format.  You  have  the  option  to 
discontinue  participation  at  any  time  without  any  adverse  action.  It  should  take  30  minutes  or  less. 

This  research  is  based  on  the  2006-2007  Competency  Model  developed  for  the  Office  of  the  Under 
Secretary  of  Defense  Acquisition,  Technology,  and  Logistics  (OSD(AT&L));  sponsored  by  HQ  Air  Force 
Material  Command  (AFMC/A5S)  in  alignment  with  ongoing  Strategic  Workforce  Management 
Initiatives;  and  endorsed  by  AFRL/DP.  The  results  are  broadly  relevant  to  workforce  development  & 
certification,  education  &  training  curriculum,  and  assignment  considerations;  they  are  specifically 
relevant  to  the  Individual  Development  Plans  (IDPs)  of  S&T  employees  and  then  supervisors. 

Instructions 

Please  complete  the  experience  data  in  Part  I.  In  Parts  II  &  III  you  will  be  asked  to  review  the 
definitions/behaviors  for  each  technical  competency  (36)  and  professional  competency  (27)  and  then 
select  a  response  to  the  three  same  questions  about  each  competency.  Part  IV  includes  three  open-ended 
questions;  you  may  use  as  much  space  as  necessary  to  explain  your  responses. 

I.  TOTAL  EXPERIENCE  in  DoD  S&T  (mark  all  that  apply;  highlight  most  recent  status  held) 

As  a  Military  Member 


Years  Experience 

Managing  S&T/Supervising  STMs 

<  2  Years 

3-6  Years  7-12  Years  >12  Years 

As  a  Government  Civilian 

Years  Experience 

Managing  S&T/Supervising  STMs 

<  2  Years 

3-6  Years  7-12  Years  >  12  Years 

As  a  Government  Contractor 

Years  Experience 

Managing  S&T/Supervising  STMs 

<  2  Years 

3-6  Years  7-12  Years  >12  Years 

APDP  Certifications:  (indicate  all  areas  that  apply) 

In  S&T  management.  I’m  considered: 

Level  I: 

Entry  Level  (Beginner) 

Level  II: 

Journeyman  Level  (Intermediate) 

Level  III: 

Expert  Level  (Senior) 

Degree(s):  Engineering 

Science 

Mathematics 

Other(s) 

Years  in  AFRL  Orgs  (mark  all  that  applv:  highlight  most  recent):  RD  :  RH  :  RI 

RX  ;  RW  ;  RZ  ;  RY 

;  RB 

;  RV  ;  AFOSR  :  HQ 

Other  S&T-affiliated  Org(s): 

COMPETENCY  QUESTIONS  1/20 
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II.  TECHNICAL  COMPETENCIES 


1.  Requirements  Process:  (a.)  identify  /  analyze  /  prioritize  needs:  prepare  a  Functional  Needs  Analysis  to  compare 
alternatives,  identify  capability  gaps  &  redundancies,  reflect  inefficiencies.  &  address  priorities  while  defining  an 
integrated  architecture  of  multiple  views  which  facilitate  integration  and  promote  interoperability  across  capabilities 
and  among  related  integrated  architectures;  (b.)  define  user  needs  and  constraints:  to  include  performance 
parameters,  objectives,  &  thresholds;  affordability,  scheduling,  &  technical  constraints:  and  environmental  issues 
while  performing  tradeoffs 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never:  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 


2.  Technology  Development  Strategy  /  Acquisition  Strategy:  (a.)  technology  development,  acquisition,  & 

management  strategies:  prepare  a  Technology  Development  Strategy  for  milestone  approval  authority  no  later  than 
the  completion  of  a  Concept  Refinement  Phase;  prepare  an  Acquisition  Strategy  for  milestone  approval  authority  no 
later  than  the  completion  of  a  Technology  Development  Phase  to  include  a  plan  for  total  life-cycle  system 
management,  contracting,  and  sustainment;  (b.)  conduct  project/program  coordination  with  users,  MDAs.  industry, 
etc.;  (c.)  track  development  of  AoA  exploring  numerous  conceptual  solutions  towards  concept  refinement  phase  and 
technology  development  strategy  with  justification  for  formal  initiation  of  acquisition  program;  (d.)  ensure 
development  of  business  partnerships  between  PM  and  PCO  built  on  a  successful  acquisition  strategy 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

I  low  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entiy;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 


3.  Core  Management  Skills  &  Processes:  (a.)  Phase  Inputs/Outputs:  Management  Reviews:  plan  program  scope. 

security  measures,  total  Life-cycle  system  management  addressing  scheduled  “Inputs”  and  “Outputs”  for  each  phase 
to  inchtde  internal  &  external  project/program  management  reviews:  (b.)  Project  Management  Master  Schedule: 
prepare/document  progr  am  schedule  via  technology/methods  using  portfolio  &  project  management  software; 

(c.)  WBS:  prepare  program  &  contract  Work  Breakdown  Structures  (WBS)  and  integrate  with  other 
processes/documents;  (d.)  Technical  Reviews  &  Tradeoffs:  conduct  technical  reviews  and  perform  trade-offs: 

(e.)  EVM  Baseline.  Six  Sigma,  Lean:  ov  ersee  &  coordinate  Earned  Value  Management  (EVM)  baseline  review 
processes,  strategies,  negotiations,  while  applying  “lean”  and  “six  sigma"  skills  as  appropriate 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somew'hat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  w’ork? 

(1)  Never"  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  wrhat  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


COMPETENCY  QUESTIONS  2/20 
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4.  Life-Cycle  Cost  (Total  Ownership  Cost)  Management:  (a.)  prepare  an  estimate  of  TOC  applying  basic  cost¬ 
estimating  techniques/tools  regarding  management  decisions;  (b).  develop/submit  out-year  financial  plans  &  budget 
estimated  in  DoD  format  based  on  EVM  processes 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Veiy  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman:  (3)  Senior/Expert;  N/A 

5.  Risk  Management:  (a.)  conduct  risk  assessments,  identify  risk  events,  perform  risk  analysis:  monitor/report  risk 
status:  integrate  risk  management  into  routine  management  practices  (b.)  apply  decision  analysis  hi 
selecting/prioritizing  risk  handling  options  folded  into  a  detailed  Integrated  Master  Plan  (IMP);  (c.)  employ  risk 
management  software  for  tracking,  rating.  &  handling  risk  events,  identifying  critical  paths,  and  determining 
probabilities  of  progr  am  completion  dates  and  costs 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Veiy  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry:  (2)  Journeyman;  (3)  Senior/Expert:  N/A 

6.  Joint/Cross  Agency /International  Program  Management  by  single  U.S.  Executive  Agency:  formulate,  plan, 
implement,  manage,  track  and  evaluate  joint,  interagency  acquisition  projects/programs 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never:  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

7.  Market  Research:  develop  a  market  research  strategy  per  Federal  Acquisition  Regulation  (FAR)  Part  10  and 
extends  to  dual-use  technologies  as  applicable 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Veiy  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman:  (3)  Senior/Expert;  N/A 

COMPETENCY  QUESTIONS  3/20 
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8.  Communications  Management  and  the  IPTIPPT  Process:  (a.)  Effective  Communications  &  Media:  employ 
oral/written  skills  for  all  intemal/extemal  briefings,  communicating  lessons  learns,  and  application  of  public  affairs 
policies;  (b.)  employ  IPPD  &  IPT  Process:  maintaining  open  communication  processes  with  staff,  contractors,  & 
supporting  agencies  while  reinforcing  team  building,  leadership,  and  empowerment;  apply  metrics  principles  of 
change  management 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How"  often  is  this  competency  used  hi  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


9.  Configuration  Management,  Data  Management  and  Information  Management:  utilize  knowledge  of 

(a.)  principles/methods  for  planning/managing  the  implementation,  update,  &  integration  of  information  system 
components;  (b.)  data  management  tools  such  as  modeling,  backup,  recovery,  warehousing,  mining,  disposal,  and 
data  standardization  processes;  (c.)  where/how  to  gather,  organize,  &  maintain  information  management  systems 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important;  (5)  Extremely  Important 
How"  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


10  Information  Resource  Strategy  and  Planning,  System  Life-cycle,  and  Management/Techuology  Awareness: 
utilize  knowledge  of  (a.)  Capital  Planning  and  Investment  Control  including  requirements  of  CCA,  OMB  A-l  1  /  A- 
130  /  A-300  to  include  systems  life  cycle  management  concepts  used  in  planning,  developing,  implementing, 
operating,  and  maintaining  information  systems:  (b.)  IM/IT  Methods.  Assessment.  Development:  principles, 
methods,  &  techniques  of  IM/IT  assessment,  planning,  management,  monitoring,  &  evaluation  such  as  baseline 
assessment,  interagency  functional  analysis,  contingency  planning,  and  disaster  recovery;  (c.)  new  developments  & 
applications  of  IM  /  IT  (policies,  processes,  methods,  hanhvare,  software,  and  telecommunications) 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somew  hat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job1 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 


COMPETENCY  QUESTIONS  4/20 
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11  Information  Systems:  Network  Security/Assurance,  Architecture.  Performance.  Infrastructure  Design  and 
System  Management:  utilize  knowledge  of  (a.)  IM  /  IT  certification  and  accreditation  methods,  tools,  & 
procedures  for  developing  information  security  system  plans  to  prevent  system  vulnerabilities  and  provide/restore 
security  of  information  systems  and  network  sendees;  (b.)  IM  /  IT  systems  engineering  to  include  architecture 
methodologies  design  and  development  along  with  typology  of  software,  hardware,  &  network  protocols  and 
standards  integrated  with  one  another  and  associated  controlling  software;  (c.)  IM  /  IT  performance  measures  and 
service-level  agreements  (SLA)  to  assess  the  quality,  effectiveness,  &  practicality  of  IT  systems 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important:  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


12  System  Integration:  utilize  knowledge  of  the  principles,  methods,  and  procedures  for  installing,  integrating,  & 
optimizing  information  systems  components  including  knowledge  of  system  test  &  evaluation  and  software 
verification  and  validation  for  large-scale  IM  /  IT  procurements 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important:  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


13  Technical  Management  Process:  (a.)  Tech  Plan/Assessment,  Systems  Engineering  Methods:  prepare/apply 
decision  analysis  method,  technical  plans  &  assessment,  requirements  management  method,  and  technical  data 
management  plan  for  selecting  alternatives,  applying  systems  engineering  processes  properly,  conducting  technical 
reviews,  providing  traceability  back  to  user-defined  capabilities,  &  acquiring/accessing  technical  data;  (b.)  Integrate 
Risk  Management  Plan:  integrate  risk  management  plan/methods  for  systems  engineering  context  examining  risks  of 
deviating  from  program  plan  design  performance  requirements  with  respect  to  other  lifecycle  issues  such  as 
manufacturing,  operations,  environment,  safety,  occupational  health  considerations.  &  support:  (c.)  Apply  Config 
Management  Methods:  apply  configuration  management  methods/practices  for  consistency  of  a  product's  attributes, 
resulting  in  a  complete  audit  trail  of  decisions  and  design  modifications;  (d.)  Interface  Management:  prepare/apply 
process  for  interface  management  to  ensure  interface  definition  &  compliance  among  system  elements  and  other 
systems 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Veiy  Frequently 

At  w'hat  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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14.  Technical  Process:  (a.)  Working  with  the  User:  prepare  &  apply  a  requirements  development  process  for  working 
with  the  user  to  establish  &  refine  operational  needs,  attributes,  performance  parameters.  &  constraints  that  flow 
from  needed  capabilities,  then  ensure  all  relevant  requirements  are  addressed;  (b.)  Developing  Logical  Analysis 
Process:  prepare/apply  a  logical  analysis  process  to  obtain  sets  of  logical  solutions  to  improve  understanding  of  the 
defined  requirements  &  their  relationships;  prepare/apply  a  process  for  monitoring  &  selecting  design  solutions  that 
translate  outputs  of  the  requirements  development  and  logical  analysis  process  into  alternative  design  solutions;  (c.) 
Monitoring  Physical  Architecture:  prepare/apply  process  for  monitoring  the  Implementation  process  that  actually 
yields  the  lowest  level  system  elements  in  the  system  hierarchy,  then  incorporating  the  lower  level  system  elements 
into  a  higher-level  system  element  in  the  physical  architecture:  (d).  Verifying  Elements  Meet  Specifications: 
prepare/apply  a  process  to  monitor  the  verification  process  to  confirm  systems  element  meets  the  design-to/build-to 
specifications,  then  participate  in  the  validation  process:  (e).  Transitioning  Element  to  Next  Level:  prepare/apply  a 
process  to  move  the  system  element  to  the  next  level  in  the  physical  architecture,  or  for  end-item  system,  to  the  user 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

15.  Systems  Engineering  Plan  (SEP):  prepare  and  maintain  a  Systems  Engineering  Plan  which  will  describe  required 
engineering  activities  and  address  the  who.  w  hat,  when,  where,  why.  and  how  of  the  applied  systems  engineering 
approach,  including  security 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry:  (2)  Journeyman:  (3)  Senior/Expert;  N/A 

16  Software  Development.  Acquisition  Management  Technical  Fundamentals.  Quality  and  Measurement: 

(a.)  Software  Development  Plan:  plan,  select,  apply  &  conduct  a  development  process,  key  plans/approaches,  life- 
cycle  reviews,  and  development  methodologies:  select  programming  languages  and  interpret/apply 
commercial/govemmental  off-the-shelf  methods:  (b.)  Software  Fundamental  Application:  apply  software 
engineering  fundamentals  in  the  areas  of  architecture,  hardware,  COTS/ERP  management  processes, 
communications,  networking,  quality  assurance  methods,  error  density  methods,  and  integration 
verification/validation  methods:  plan,  develop.  &  employ  software  performance,  management.  &  technical 
measurement  methods,  including  capability  maturity  models  &  measurement  examples 

What  is  die  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  pomt  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A _ 

COMPETENCY  QUESTIONS  6/20 


77 


17  Software  Process  Maturity,  Critical  Requirements,  Data  Management,  Software  Support  and  Safety:  plan, 
select,  &  apply  process  maturity  models  and  maturity  return  on  investment  (ROI)  methods:  select  safety,  privacy  & 
security  requirement  methods:  apply  data  &  net-centric  data  management  process:  develop  support  characteristics, 
support  plan.  &  life-cycle  plan:  plan  &  apply  safety  issues,  procedures,  &  tools 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never:  (2)  Sometimes:  (3)  Often:  (4)  Frequently:  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry:  (2)  Journeyman:  (3)  Senior/Expert:  N/A 

18  Software  Reliability,  Reuse,  and  SIS  Independent  Expert  Reviews:  regarding  software  reliability,  select  and 
apply  reliability  measurement  methods,  software  RAM  issues,  &  models:  develop  reuse  considerations  and  discuss 
repositories:  organize  SIS  expert  reviews 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important:  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never:  (2)  Sometimes:  (3)  Often:  (4)  Frequently:  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A _ 

19  S&T  Goal,  Program  Considerations,  aud  Technology  Engineering:  (a.)  S&T  Future  Needs:  investigate  &  track 
user  needs:  construct  &  plan  a  broad-based  program  spanning  all  Department/Agency  relevant  sciences  & 
technologies  to  anticipate  future  needs  and  those  not  being  pursued  by  civil  or  commercial  communities  while 
planning  to  ensure  the  conduct  of  S&T  activities  do  not  preclude/facilitate  future  competition:  (b.)  S&T  Transition 
to  Operational  System:  develop  a  program  of  long-range  research  regarding  the  implementation  and  transition  of 
S&T  and  new  methods  into  operational  systems:  gather  &  assess  opportunities  &  evaluation  of  technical  readiness 
for  operational  application 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important:  (4)  Veiy  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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20.  Transition  Techniques:  (a.)  Anticipate  Future  S&T  Needs:  address  user  needs;  maintain  a  broad-based  program 
spanning  all  relevant  sciences  and  technologies  to  anticipate  future  needs  and  those  not  being  pursued  by  civil  or 
commercial  communities;  (b.)  S&T  Transition  to  Operational  System:  develop  a  program  of  long-range  research, 
the  implementation  &  transition  of  science,  technology.  &  new  methods  into  operational  systems;  gather  &  assess 
opportunities  &  evaluation  of  technical  readiness  for  operational  application 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 


21  Identify  and  Protect  Promising  Technologies;  Apply  a  process  for  reducing  the  security  risks  of  introducing 
these  technologies  into  the  acquisition  process  in  laboratories  and  research  centers,  academia,  and  foreign  & 
domestic  commercial  sources 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  pomt  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


22  Test  &  Evaluation  (T&E)  Integration,  Strategy,  and  Planning:  (a.)  T&E  Plan:  develop  a  robust,  integrated 
T&E  strategy  for  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E):  develop  a 
vulnerability  test  to  validate  system  performance  and  ensure  the  product  provides  measurable  improvement  to 
operational  capabilities;  (b.)  Event  Driven,  Realistic  Testing:  integrate  modeling  and  simulation  (M&S)  activities 
with  government  and  contractor  DT&E,  OT&E,  vulnerability,  interoperability.  &  performance  testing  into  an 
efficient  continuum  in  concert  with  the  user  &  test  communities  without  compromising  rigor;  ensure  testing  is  event 
driven  within  the  program’s  overall  acquisition  strategy;  prepare  T&E  Report  preparation  and  document  these  plans 
(coordinating  with  Acquisition  Strategy)  in  comprehensive  T&E  Maser  Plan  (TEMP),  including  security; 

(c.)  Minimize  Test  Cost:  integrate  the  many  aspects  of  T&E  to  optimize  test  scope  and  minimize  cost 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  w'ork  function  to  perform  your  job? 

1)  Entry:  (2)  Journeyman;  (3)  Senior/Expert:  N/A 
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23  Development  Test  &  Evaluation  (DT&E):  develop  test  to  validate  systems  engineering;  ensure  Development 
Test  will  provide  verification  and  validation  of  the  systems  engineering  process  and  confidence  that  the  system 
design  solution  is  on  hack  to  satisfy  the  desired  capabilities;  ensure  T&E  strategy/plan  is  consistent  with  and 
complementary  to  the  Systems  Engineering  Plan 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important;  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


24  Operational  Test  and  Evaluation  (OT&E):  achieve  maturity  prior  to  Initial  OT&E:  apply  department/agency 
process  for  IOT&E  to  preclude  the  item/system  from  entering  IOT&E  prematurely  by  ensuring  it  has  demonstrated 
technical  maturity  under  the  conditions  expected  in  the  IOT&E;  ensure  final  OT  tests  include  vulnerability  testing 
and  that  the  tests  are  conducted  under  realistic  conditions  to  determine  effectiveness  &  suitability  of  the  item 


What  is  the  degr  ee  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job1 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


25  Life-Cycle  Logistic  Management.  Product  Support,  aud  Interoperability:  (a.)  Life-Cycle  Product  Support: 
examine  &  implement  innovative,  alternative  logistics  support  practices,  including  best  public  sector  &  commercial 
practices  and  technology  solutions;  establish  logistics  support  program  goals  for  cost,  customer  support,  and 
performance  parameters  over  the  program  life  cycle;  include  as  part  of  the  Acquisition  Strategy  a  program  manager 
developed  sustainment  strategy  for  Life-Cycle  Product  Support;  (b).  Open  Systems  and  Cross-Servicing:  develop  a 
modular  open  systems  approach  (MOSA)  where  interoperability  is  a  key  LCL  facilitator,  allowing  the  program 
manager  to  take  advantage  of  shared  government-wide  capabilities  in  designing  and  implementing  a  product  support 
strategy;  consider  long-term  potential  of  Acquisition  and  Cross-Servicing  Agreements  (ACSAs):  apply  logistic  risk 
mitigation  analysis  early  in  system  development  to  reduce  required  resources  and  overall  life  cycle  costs 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  hr  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


COMPETENCY  QUESTIONS  9/20 


80 


81 


28  Contract  Approach,  Requirements  and  Supporting  Documents.  Prepare  &  Issue  Solicitation:  (a.)  Business 
Partnership  &  Contract  Plan:  apply  a  process  by  which  the  efforts  of  the  PM,  PCO.  and  all  other  personnel 
responsible  for  contractual  matter  are  integrated  through  a  comprehensive  plan  for  fulfilling  agency  needs  in  a 
timely  manner  and  at  a  reasonable  cost;  develop  overall  contract  strategy  for  managing  the  acquisition,  coordination 
and  development  of  the  acquisition  plan,  including  support  of  the  exit  criteria  for  each  acquisition  phase;  develop  a 
business  partnership  between  the  PM  and  PCO  emphasizing  successful  acquisition  sttategy  leading  to  program 
success;  ensure  potential  and  actual  contractors,  sub-contractors,  &  affiliated  government  organizations/offices  have 
full  comprehension  of  program  definition  plus  procuring  agency’s  organizational  culture  and  organizational 
structure;  (b.)  Plan  Solicitation:  apply  pre-award  policies  in  accordance  with  Federal  Acquisition  Regulation  (FAR) 
and  key  contr  act  terms  &  conditions  for  solicitation;  (c.)  Pre-Award,  SOW,  Roles  &  Missions:  apply  pre-awar  d 
policies  per  FAR;  manage  preparation  of  comprehensive  program  specification  and  statement  of  work  (SOW)  that 
fully  &  correctly  defines  the  program,  addressing  roles  &  missions  of  the  government  and  contractor 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry:  (2)  Journeyman:  (3)  Senior/Expert;  N/A 


29  Sour  ce  Selection,  Contract  Award,  Contract  Administration.  &  Contract  Close-out:  (a.)  Source  Selection: 
apply  support  source  selection  criteria  per  FAR  Part  15;  (b.)  Awards  Process:  support  contract  award  process  per 
FAR  Part  1 5  -  Contracting  by  Negotiation  /  PreAward,  Award.  &  Post  Award  Notifications,  Protests.  &  Mistakes; 
(c.)  Contr  act  Administration:  support  contract  administrative  actions  per  FAR  Part  42  -  Contract  Administration  and 
Audit  Services,  while  addressing  “base-lining”  the  contract  as  in  Research  and  Technology  Protection  (RTP)  actions 
and  outlining  the  contracting  officer  representative  (COR)  and  PM  duties  for  administering  contract  requirements — 
e.g.  contract  modifications  process,  receipt  of  contractor  change  proposals  and  contractor  financing  requirements; 
(d.)  Termination:  support  procedures  for  contract  closeout  per  FAR  Part  49  -  Termination  of  Contracts 


What  is  tlie  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How7  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 


(1)  Entry:  (2)  Journeyman:  (3)  Senior/Expert;  N/A 


30  Performance-based  Service  Agreements:  establish  a  negotiated  baseline  of  performance  w  ith  system  users,  and 
the  corresponding  support  necessary  to  achieve  that  performance,  from  both  commercial  and/or  organic  support 
providers;  utilize  the  baseline  performance  and  negotiate  the  required  level  of  support  with  the  providers  to  achieve 
the  desired  performance  at  a  cost  consistent  with  available  support  funding 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How7  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Fxequently:  (5)  Very  Frequently 

At  what  point  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 
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31  Business  Financial  Planning  &  Management  and  Cost  Estimating:  (a.)  Total  Cost  /  Performance  / 
Affordability:  apply  Total  Life  Cycle  Systems  Management  (TLCSM)  which  requires  die  PM  to  base  major 
decisions  on  system-wide  analyses  and  the  lifecycle  consequences  of  those  decisions  and  on  system  performance  & 
affordability:  (b.)  Cost  Estimating  Skills:  apply  cost  estimating  processes,  methods,  techniques,  analytical  principles, 
data,  specialized  costing,  reconciliation  processes  with  other  estimates,  and  management  applications 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important:  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 


32  Earned  Value  Management  (EVM):  (a.)  Apply  EVM:  apply  EVM  management  policies,  methodologies.  & 
software  for  performance  measurement  of  programs  including:  Technical  Performance  Measurement  selection  and 
tracking  vs.  scheduled  data  collection  events:  (b.)  Apply  Baseline  Review:  apply  the  Integrated  Baseline  Review 
(IBR)  process;  (c.)  Prepare  Corrective  Strategies:  apply  analytical  &  evaluation  techniques  to  determine  effective 
program  strategies  when  EVM  indicators  are  yellow  and/or  red 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Veiy  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


33  Fiuaucial  Reporting  &  Oversight  aud  Programming,  Plauning.  &  Budgeting  System:  (a.)  Employ  Financial 
Management  Systems:  select  &  employ  financial  information  system(s)  in  analysis  tasks  &  report  preparation 
intended  for  collecting,  processing,  maintaining,  transmitting,  and  reporting  data  about  financial  events  in  support  of 
financial  planning  or  budgeting  systems/documents;  report  cost  information  or  support  the  preparation  of  financial 
statements;  (b.)  Agency  Financial  Management  System:  apply  department/agency’s  policy/instructions  for  financial 
planning,  programming,  budget  development  &  execution  on  a  managed  project/program  that  supports  the 
development  &  decision  making  towards  meeting  internal/external  financial  management  obligations 


What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  pomt  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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34  Industrial  Base  Assessment:  know  manufacturing  resources:  analyze  international  and  domestic  sources  which 
can  meet  the  required  need  as  the  primary  sources  of  supply  (consistent  with  relevant  domestic  preference  statutes 
per  FAR  Part  25  —  Foreign  Acquisition) 

What  is  the  degree  of  impact  tins  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  w  hat  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

35  Plan  Production:  apply  pre-production  management  practices  that  will  establish  if  the  item/system  has  achieved 
the  following:  acceptable  performance  in  development:  T&E  and  operational  assessment;  a  mature  softw'are 
capability;  no  significant  manufacturing  risks;  manufacturing  processes  under  control:  acceptable  interoperability; 
acceptable  operational  supportability;  and  demonstration  the  system  is  affordable  throughout  the  life  cycle, 
optimally  funded,  and  properly  phased  for  rapid  acquisition 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important:  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  wrork  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

36  Produce  Product:  (a.)  Low-Rate  Production  Decision:  apply  management  practices  leading  to  an  adequate  & 
efficient  manufacturing  capability  and  production  (if  applicable)  of  the  minimum  quantity  necessary  to  provide 
production  or  production-representative  articles  for  IOT&E;  establish  an  initial  production  base  for  the  system; 
permit  an  orderly  increase  in  the  production  rate  for  the  system,  sufficient  to  lead  to  full-rate  production  upon 
successful  completion  of  operational  and  vulnerability  testing,  where  applicable:  (b.)  Standards  for  Production: 
apply  appropriate  recognized  standards  to  evaluate  product  performance,  i.e.  NIST,  ISO,  ANSI 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important:  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  wrork  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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II.  PROFESSIONAL  COMPETENCIES 


1.  Accountability:  Holds  self  and  others  accountable  for  measurable  high-quality,  timely,  and  cost-effective  results: 
determines  objectives,  sets  priorities,  and  delegates  work;  accepts  responsibility  for  mistakes;  complies  with 
established  control  systems  and  rules 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

2.  Conflict  Management:  anticipates  and  takes  steps  to  present  counter-productive  confrontations:  manages  and 
resoles  conflicts  and  disagreements  in  a  constructive  manner 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  wTork  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman:  (3)  Senior/Expert:  N/A _ 

3.  Continual  Learning:  assesses  and  recognizes  own  strengths  and  weaknesses:  pursues  self-development 
What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently:  (5)  Very  Frequently 

At  w  hat  point  in  your  career  did  you  FIRST  use  this  w  ork  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman:  (3)  Senior/Expert;  N/A 

4.  Creativity  and  Innovation:  develops  new'  insights  into  situations:  questions  conventional  approaches:  encourages 
new  ideas  and  innovations:  designs  and  implements  newr  or  cutting  edge  programs/processes 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important:  (4)  Very  Important:  (5)  Extremely  Important 

How-  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  wrork  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A _ 
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5.  Customer  Service:  anticipates  and  meets  the  needs  of  both  internal  and  external  customers;  delivers  high-quality 
products  and  services;  is  committed  to  continuous  improvement 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  hi  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

6.  Decisiveness:  makes  well-informed,  effective,  and  thnely  decisions,  even  when  data  are  limited  or  solutions  produce 

unpleasant  consequences;  perceives  the  impact  and  implications  of  decisions _ 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  hi  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  pohit  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

7.  Developing  Others:  develops  the  ability  of  others  to  perform  and  contribute  to  the  organization  by  providing 
ongoing  feedback  and  by  providing  developmental  opportunities  to  learn  through  formal  and  informal  methods 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  hi  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  pohit  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

8.  Entrepreneurship:  positions  the  organization  for  future  success  by  identifying  new  opportunities:  builds  the 
organization  by  developing/improving  products  or  sendees;  takes  calculated  risks  to  meet  organizational  objectives 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  ExUemely  Important 

How  often  is  this  competency  used  hi  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  pohit  hi  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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9.  External  Awareness:  understands  and  keeps  up-to-date  on  local,  national,  and  international  policies/trends  that 
affect  the  organization  and  shape  stakeholders'  views;  is  aware  of  the  organization’s  impact  on  external  environment 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extr  emely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 

10.  Financial  Management:  understands  the  organization's  financial  processes:  prepares,  justifies,  and  administers 
program  budget;  oversees  procurement  and  contr  acting  to  achieve  desired  results;  monitors  expenditures  and  uses 
cost-benefit  thinking  to  set  priorities 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

1 1 .  Flexibility:  is  open  to  change  and  new  information;  rapidly  adapts  to  new  information,  changing  conditions,  or 
unexpected  obstacles 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

12.  Human  Capital  Management:  builds  and  manages  workforce  based  on  organizational  goals,  budget 
considerations,  and  staffing  needs;  ensures  that  employees  are  appropriately  recruited,  selected,  appraised,  and 
rewarded;  takes  action  to  address  performance  problems;  manages  a  blended  workforce  and  variety  of  situations 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  per  forming  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 
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13.  Influencing/Negotiating:  persuades  others;  builds  consensus  through  give  and  take;  gains  cooperation  from  others 
to  obtain  information  and  accomplish  goals 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Veiy  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

14.  Interpersonal  Skills:  treats  others  with  courtesy,  sensitivity,  and  respect:  considers  and  responds  appropriately  to 
the  needs  and  feelings  of  different  situations 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

15.  Leveraging  Diversity:  fosters  an  inclusive  workplace  where  diversity  and  individual  differences  are  valued  and 
leveraged  to  achieve  the  vision  and  mission  of  the  organization 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

16.  Oral  Communication :  makes  clear  and  convincing  oral  presentations:  listens  effectively:  clarifies  information  as 
needed 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman:  (3)  Senior/Expert;  N/A 
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17.  Partnering:  develops  networks  and  builds  alliances;  collaborates  across  boundaries  to  build  strategic  relationships 
and  achieve  common  goals 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Vety  Important:  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

18.  Political  Savvy:  identifies  the  internal  and  external  politics  that  impact  the  work  of  the  organization:  perceives 
organizational  and  political  reality  and  acts  accordingly 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

19.  Problem  Soil  ing:  identifies  and  analyzes  problems:  weighs  relevance  and  accuracy  of  information;  generates  and 
evaluates  alternative  solutions;  makes  recommendations 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 

20.  Public  Service  Motivation:  shows  a  commitment  to  serve  the  public:  ensures  that  actions  meet  public  needs;  aligns 
organizational  objectives  and  practices  with  public  interests 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert:  N/A 
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21.  Resilience:  deals  effectively  with  pressure:  remains  optimistic  and  persistent,  even  under  adversity;  recovers  quickly 
from  setbacks 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

22.  Strategic  Thinking:  formulates  objectives,  priorities,  and  plans  consistent  with  the  long-term  business  and 
competitive  interests  of  the  organization  in  a  global  environment;  capitalizes  on  opportunities  and  manages  risks 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  wrork? 

(1)  Never;  (2)  Sometimes;  (3)  Often:  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

23.  Team  Building:  inspires  and  fosters  team  commitment,  spirit,  pride,  and  trust;  facilitates  cooperation  and  motivates 
team  members  to  accomplish  group  goals 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important:  (5)  Extremely  Important 
How  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

24.  Technical  Credibility:  understands  and  appropriately  applies  principles,  procedures,  requirements,  regulations,  and 
policies  related  to  specialized  expertise 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

Howr  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry:  (2)  Journeyman;  (3)  Senior/Expert:  N/A 
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25.  Technology  Management:  keeps  up-to-date  on  technological  development;  makes  effective  use  of  technology  to 
achieve  results;  ensures  access  to  and  security  of  technology  systems 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Very  Important;  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  w  ork? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


26.  Vision:  takes  a  long-term  view  and  acts  as  a  catalyst  for  organizational  change:  builds  a  shared  vision  with  others; 
influences  others  to  translate  vision  into  action 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important;  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes;  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 


27.  Written  Communication:  writes  in  a  clear,  concise,  organized,  and  convincing  maimer  for  the  intended  audience 

What  is  the  degree  of  impact  this  competency  has  on  job  performance? 

(1)  Not  Important:  (2)  Somewhat  Important;  (3)  Important;  (4)  Veiy  Important;  (5)  Extremely  Important 

How'  often  is  this  competency  used  in  performing  work? 

(1)  Never;  (2)  Sometimes:  (3)  Often;  (4)  Frequently;  (5)  Very  Frequently 

At  what  point  in  your  career  did  you  FIRST  use  this  work  function  to  perform  your  job? 

(1)  Entry;  (2)  Journeyman;  (3)  Senior/Expert;  N/A 

IV.  OPEN-ENDED  QUESTIONS: 

(1)  Are  there  any  competencies  missing  from  this  set  that  you  think  are  critical  to  managing  S&T  programs?  If  so,  what? 


(2)  Could  demonstr  ated  proficiency  in  a  set  of  STM  competencies  serve  as  hiring  evaluation  criteria?  Why/Why  Not? 


(3)  Is  a  technical  degree  (e.g.  science,  engineering,  mathematics)  critical  to  proficient  S&T  management?  Why/Why  Not? 
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Appendix  B:  S&T  Manager  (STM)  Course  Competencies 


STM  201  -  INTERMEDIATE  S&T  MANAGEMENT  COURSE _ 

1 .  Assess  the  Science  and  Technology  Manager  Career  Path  requirements 

2.  Explain  the  Defense  Systems  Acquisition  Framework  with  regard  to 
technology  transition. 

3.  Summarize  the  impact  of  the  business  environment  on  technology  transition 

4.  Given  specifics  of  critical  technologies,  classify  them  according  to  the  nine 
levels  defined  in  the  Technology  Readiness  Levels  (TRL) 

5.  Assess  the  Future  Naval  Capabilities  process 

6.  Compare  the  various  technology  transition  processes 

7.  Explain  the  approach  used  by  the  Army  to  transition  technology 

8.  Summarize  DARPA’s  role  in  technology  transition 

9.  Analyze  the  benefits  of  the  Applied  Technology  Council  approach  to 
technology  transition 

10.  Discus  the  role  of  the  DoD  Office  of  Technology  Transition 

1 1 .  Discriminate  between  industry  and  government  mechanisms  to  transition 
technology 

12.  Develop  a  technology  transition  checklist 

13.  Apply  effective  technology  transition  practices 
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STM  302  -  ADVANCED  S&T  MANAGEMENT  COURSE 


1 .  Identify  and  explain  the  primary  objectives  of  each  phase  and  milestone  of  the 
DoD  Acquisition  Process  Model 

2.  Explain  the  principles  of  Science  &  Technology  transition,  the  acquisition 
lifecycle,  total  ownership  costs,  the  S&T  -  acquisition  interface,  and  S&T 
transition  management  objectives 

3.  Demonstrate  an  understanding  of  the  technology  engineering  management 
process  to  create  Defense  Capabilities  for  existing  and  future  requirements 

4.  Develop  integrated  architectures  for  DoD  systems  and  understand  the 
interoperability  certification  process 

5.  Given  an  acquisition  scenario  within  the  IPPD  environment,  the  student  will 
be  able  to  develop  and  present  the  outputs  of  the  systems  engineering  process 

6.  Given  an  acquisition  scenario  within  the  IPPD  environment,  the  student  will 
be  able  to  identify  the  key  activities  necessary  to  implement  the  systems 
engineering  process 

7.  Identify  the  benefits  and  pitfalls  in  international  acquisition  from  an  S&T 
manager’s  perspective 

8.  Evaluate  organization,  communication  and  teaming  techniques  that  facilitate 
Integrated  Product  and  Process  Development  in  the  Science  &  Technology 
program  environment 

9.  Given  a  technology  program  scenario,  develop  requirements  and  metrics  for 
managing  the  team,  affordability,  technology,  cost  &  schedules 

10.  Given  an  overview  of  alternative  evaluation  techniques,  identify  their 
opportunities  and  potential  value  for  use  in  Technology  project  management 

1 1 .  Prepare  for  the  acquisition  of  a  Software  Intensive  System  by  understanding 
the  lessons  learned,  the  government  regulations  and  guidelines,  and  the 
relevant  system  definitions 
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12.  Given  a  notional  software-intensive  system,  institute  appropriate  software 
management  plans  using  the  “16  Best  Practices”  tenets  to  address 
AT&L/S&T  Software  Intensive  Systems  (SIS)  management  concerns 

13.  Given  a  requirement  to  acquire  a  new  start  S&T  software-intensive 
system,  students  will  be  able  to  determine  the  ability  of  contractors  to 
provide  on-time  within  budget  systems  containing  high  quality  mature 
software 

14.  Given  a  scenario,  the  student  will  correctly  distinguish  the  role  of  Test  & 
Evaluation  in  the  acquisition  and  systems  engineering  processes) 

15.  Apply  the  DoD  Test  and  Evaluation  process  to  S&T  programs  and 
contribute  to  the  development  of  test  and  evaluation  master  plans  in  a  test 
IPT  environment 

16.  Identify  a  Test  &  Evaluation  strategy  for  alternative  acquisitions,  such  as 
Non-Developmental  Items  (NDI),  Commercial  Items  &  non-traditional 
acquisitions  such  as  Advanced  Concept  Technology  Demonstrations 
(ACTD) 

17.  Given  a  technology  program  scenario,  develop  requirements  and  metrics 
for  managing  the  team,  affordability,  technology,  cost  &  schedule 
activities 

18.  Analyze  key  issues  related  to  transitioning  technology  to  acquisition 
programs,  evaluate  alternative  methods  to  address  these  issues  and 
recommend  steps  that  will  lead  to  success 
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Appendix  C:  Comments  to  Open-Ended  Question  #1 


“Are  there  competencies  missing  from  this  set  that  you  think  are  critical  to  managing 
S&T  programs? 


Technical  Competencies 

•  Analysis  techniques:  statistical  process  control,  experimental  design,  forecasting, 
systems  thinking — all  tailored  to  technology-specific  S&T  fields 

•  System  of  systems  integration  and  the  development  and  use  of  architectures 

•  Reporting  scientific  results  and  applied  science:  turning  them  into  peer- 
reviewed/DTIC  reports 

•  Knowledge  of  and  familiarity  with  computer/scientific  software 

•  Operational  understanding:  development  activities  related  to  operators  in  the  field  for 
focusing  on  priorities  most  important  to  an  operator 

•  Knowledge  of  how  systems/technologies  are  used/deployed  by  users 

•  Knowledge  of  working  with  supporting  functional:  finance  (FM),  contracting  (PK), 
and  other  technology  directorate  supporting  functions 

•  Understanding  of  ethical  values  or  implications  unique  to  various  technology  areas: 
chemical/biological  weapons,  stem  cell  research,  safety  &  human  rights, 
environmental 

•  Technology-specific  expertise  unique  to  the  S&T  environment  in  which  one  is 
working 

•  Complex  thinking:  being  able  to  break  down  a  complex  issue  into  manageable  pieces 

Professional  Competencies: 

•  Marketing  skills:  advocate  for  program  development  and  technology  transition 
support  to  broad  audiences 

•  Team  integrity/trust:  sharing  the  right  information  with  the  right  people  at  the  right 
times 

•  Follow-through:  complete  agreed  actions  and  report  the  results  back  to  those  who 
need  to  know 

•  Accuracy  of  recommendations:  knowing  how  to  ensure  guidance  given  is  reliable 

•  Enthusiasm  about  the  program  being  managed:  caring  about  the  success  of  the 
program  with  a  commitment  to  delivering  results 

•  Balance:  self-awareness  of  balancing  strengths  and  weaknesses  to  avoid  extreme 
management  styles 

•  Mentorship:  how  to  mentor  and  guide  others  to  enable  them  to  better  achieve  full 
potentials  and  better  contribute  in  areas  of  inexperience 

•  Finesse:  convincing  people  to  “do  your  bidding”  even  while  they  think  they  are 
working  to  their  own  advantage 

•  Charisma:  inspiring  others  to  follow  even  through  undesirable  situations 


95 


Appendix  D:  Comments  to  Open-Ended  Question  #2 


“Could  demonstrated  proficiency  in  a  set  of  STM  competencies  serve  as  hiring 
evaluation  criteria?” 


Supportive 

•  especially  helpful  for  identifying  the  more  critical  factors  relating  to  the  specifics  of 
each  job 

•  being  able  to  ask  a  candidate  for  an  example  of  an  experience  where  they  have 
demonstrated  particular  desired  competencies  could  provide  good  indicators  of  their 
understanding  and  proficiency 

•  very  valuable  inputs  to  improve  the  hiring  process 

•  competencies  could  serve  as  a  hiring  criteria  in  order  to  get  the  right  people  into 
particular  S&T  management  jobs 

•  a  competency  model  supports  a  good  interview  process:  discerning  how  competent  an 
individual  is  in  the  areas  expected  to  perform 

•  competencies  could  and  should  be  used  in  the  hiring  process  to  help  avoid  hiring 
unqualified  managers 

•  demonstrated  proficiency  should  be  considered  a  part  of  any  hiring  activity 

•  professional  competencies  (working  with  others,  team  player,  communication  skills, 
accountability)  are  just  as  important  competencies  to  know  about  a  person  -  just  as 
important  as  technical  competencies 

•  definitely  useful,  but  the  competencies  would  need  to  be  tailored  for  the  various  S&T 
positions  (for  example,  differences  in  basic,  applied,  and  advanced  S&T  development 
phases) 

•  using  a  competency  model  could  help  in  the  “weeding  out”  process 

•  useful  for  identifying  training  &  gap-filling  needs 

•  competencies  are  useful  for  hiring  and  evaluations  at  the  mid-upper  levels,  but  would 
be  difficult  at  the  entry  levels 

•  competencies  could  be  used,  but  they  are  difficult  to  evaluation 

•  competencies  could  be  used  for  hiring,  but  not  for  evaluation 

•  competencies  could  be  used  partly,  but  not  exclusively 

•  the  application  of  a  competency  model  or  process  would  be  extremely  helpful  if  able 
to  better  identify  new  hires  who  are  already  proficient  in  key  areas 

•  it  seems  we’re  already  doing  it  this  way,  aren’t  we? 

•  useful  for  assessing  basic  PM  skills/tasks. . .  .something  like  a  basic  skills  exam 

•  integrating  competencies  into  the  interview  process  would  help  gage  a  person’s 
decisiveness,  problem  solving,  flexibility,  resiliency,  and  interpersonal  skills;  the 
technical/technology  competence  should  be  guaranteed  first 

•  useful  if  tailored  to  fit  the  specific  jobs  being  targeted 

•  the  use  of  competencies  for  hiring  /  evaluation  is  a  best  practice  rather  than  the 
current  government  standard;  it  would  be  very  useful  if  consistently  developed  and 
applied 
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•  integrating  the  professional  competencies  along  with  the  technical  competencies  is  an 
important  hiring  strategy  in  evaluating  how  well  the  candidate  would  fit  into  the 
existing  team 

•  demonstrated  competencies  should  be  one  of  the  various  criteria  to  consider  when 
hiring,  since  they  are  relevant  to  specific  job  performance 

•  demonstrated  competencies  should  be  a  consideration,  but  not  necessarily  a  firm 
requirement  since  a  very  capable  candidate  with  the  potential  to  perform  well  but  not 
proven  experience  could  be  overlooked 

•  useful  only  as  a  decision  aid  to  the  hiring  authority — not  as  a  replacement  for 
individual  judgment 

•  they  must  be  carefully  tailored  to  the  position — not  a  ‘must  have  all  or  out’ 
requirement 

•  use  of  particular  competencies  depends  on  the  program  &  position:  different 
competencies  or  degrees  of  proficiency  depending  on  basic,  applied,  or  advanced 
S&T  programs 

•  entry  levels  should  only  focus  on  technical  competencies;  advancement  into 
leadership  positions  should  hone  the  professional  competencies 

•  does  demonstrated  proficiency  mean  a  test? 

•  TRL  levels  require  different  balance  across  technical  &  professional  competencies 

•  (x5)  depends  on  the  level  of  hiring:  entry  levels  probably  don’t  have  demonstrated 
technical  competencies,  perhaps  professional  competency 


Not  Supportive 

•  competencies  are  an  overly  logic-based  approach  lacking  the  humanistic  dimension 

•  if  “demonstrated  proficiency”  equates  to  “experience,”  then  the  selection  process 
could  become  overly  rigid,  overlooking  understanding  acquired  through  academic 
exposure 

•  a  “cold”  criteria  set  could  not  be  used  to  do  final  evaluation 

•  the  Air  Force’s  track  record  on  measuring  things  like  competencies  has  been  very 
poor 

•  measuring  competencies  for  use  in  hiring  would  end  up  motivating  the  wrong 
behavior — become  a  pro  at  gaming  the  “test”  to  show  they  have  such  competencies 

•  competencies  are  too  difficult  to  evaluate  -  how  would  they  be  measured  &  applied 
consistently? 

•  (x3)  they  should  not  be  used  for  evaluation  due  to  lack  of  consistency 

•  past  (and  diverse)  job  experience  is  more  useful 
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Appendix  E:  Comments  to  Open-Ended  Question  #3 


“Is  a  technical  degree  (e.g.  science,  engineering,  mathematics)  critical  to  proficient  S&T 
program  management?” 


“Not  necessarily”  category: 

•  need  is  more  for  technical  competency  -  not  just  a  degree 

•  managing  S&T  people  -  not  really,  people  skills  are  needed 

•  not  critical;  but  helpful 

•  some  individuals  have  been  successful  in  S&T  management  without  a  technical 
degree 

•  no,  not  with  combination  of  operational  and  S&T  experience/expertise 

•  as  long  as  the  person  is  technically  competent,  a  management  degree  is  just  as 
effective  or  maybe  even  more  so  in  the  ability  to  effectively  run  a  program, 
serving  as  the  manager  and  not  the  chief  engineer 

•  picking  a  solid  team  can  overcome  any  perceived  shortfall 

•  not  if  the  individual  is  a  critical  thinker  who  can  ask  basic  questions  regarding 
purpose,  logical  approach,  and  expected  results;  would  benefit  from  having  a 
strong  individual  in  a  Tech  Advisor  role,  who  can  provide  technical  oversight  to 
ensure  quality  science  is  produced  and  especially  provide  staff  mentoring  in 
scientific  methods,  sound  research,  and  professional  association  networks 

•  no,  have  known  some  [S&T  PMs]  without  [technical  degrees]  who  are  very  good: 
ability  to  leam,  being  flexible,  and  being  persistent  in  finding  out  info  are  more 
important 

•  many  S&Es  aren’t  good  managers;  education  is  often  theory,  not  the  application 

•  no,  one  of  the  best  technical  managers  had  a  history  degree;  he  made  up  for  his 
lack  of  technical  education  with  decades  of  experience  in  the  operational  world 
doing  operational  test  and  tactics  development,  providing  him  with  the  “vision” 
for  what  was  needed  from  the  technical  people  in  his  organization. 

•  You  still  need  core  technical  excellence  in  the  organization,  but  diversity  of 
backgrounds  stimulates  creativity 

•  the  lack  of  a  technical  degree  can  sometimes  be  an  advantage:  a  PM  won’t  get 
hung  up  on  the  technical  details  and  can  instead  focus  on  other  important 
programmatic  matters 

•  a  “team”  consisting  of  a  strong  program  manager  “business  type”  combined  with 
a  strong  technical  advisor  can  be  just  as  good  if  not  better 

•  a  technical  degree  is  not  critical,  but  the  person  would  have  to  have  some 
knowledge  and  understanding  of  the  technology  being  developed  or  they  would 
need  a  strong  technical  lead  they  could  trust  implicitly  to  ensure  the  right 
technology  is  being  developed 

•  no,  it’s  more  important  to  be  able  to  understand  how  business  processes  work  and 
how  the  technology  fits 
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“Absolutely”  category: 

•  candidates  should  be  able  to  apply  scientific  methods,  understand  scientific  and 
engineering  documents,  literature,  and  practical  applications  of  engineers 

•  yes,  in  order  to  be  viewed  as  competent  as  well  as  having  some  personal  experience 
to  make  decisions 

•  yes,  without  a  solid  technical  competency  we  are  incapable  of  effectively  leading  the 
development  programs  and  are  forced  to  rely  100%  upon  the  contractor  for  critical 
technical  decisions  which  is  biased  to  benefit  the  contractor’s  financial  position 

•  technical  knowledge  of  the  particular  competency  is  critical  to  run  a  good  program 
and  get  beneficial  results  for  the  AF 

•  yes,  for  first  and  second  level  supervisors  in  order  to  properly  find  new  work,  allocate 
it  efficiently,  evaluate  its  progress,  and  make  go/no  go  decisions  on  the  direction  of 
the  effort;  “senior  leaders”  (at  the  2-ltr  level)  could  get  away  with  a  non-technical 
degree,  but  must  score  well  on  the  competencies  listed  and  should  be  surrounded  by 
expertise  with  higher  degrees  (e.g.  a  tech  advisor) 

•  yes,  S&T  management  is  very  related  to  understanding  the  underlying  science  and 
engineering  of  the  program 

•  yes,  a  technical  background  is  essential  to  being  able  to  direct  and  lead  the  highly 
technical  individuals  who  are  usually  part  of  an  S&T  program;  without  a  fundamental 
technical  understanding  (based  upon  a  tech  degree),  the  S&T  manager  will  find  it 
difficult  to  communicate  the  technical  details  to  non-tech  folks  and  will  find  it 
difficult  to  communicate  with/understand  the  S&Es  they  lead 

•  yes,  initially  to  give  the  S&T  PM  credibility  and  some  confidence  in  the  job 

•  yes,  you  need  technical  experience  to  understand  and  guide  the  work  of  others  all  the 
way  to  the  commander 

•  yes,  it  would  be  difficult  for  somebody  without  at  least  a  basic  working  knowledge  of 
technical  terms  to  effectively  manage  an  S&T  program 

•  STMs  should  not  rely  solely  on  the  judgment  of  advisors:  they  may  need  to  choose 
between  conflicting  advice  from  different  advisors  or  be  able  to  devise  their  own  best 
course  of  action;  they  also  need  to  understand  why  something  is  a  technical  challenge 
or  physical  impossibility 

•  yes,  a  technical  degree  usually  means  that  the  individual  will  be  a  good  problem 
solver  and  is  fairly  logical  in  their  thought  process 

•  yes,  a  technical  degree  is  critical  to  S&T  management  because  the  S&T  manager  has 
to  understand  the  technology  being  developed  or  risk  providing  ineffective 
management  to  the  project. 

•  yes,  for  problem  solving 

•  yes,  absolutely  in  order  to  be  technically  savvy,  ask  technical  questions  and 
understand  trades  and  problems  within  context 

•  yes,  a  technical  degree  is  essential  to  advanced  S&T  work 

•  yes,  provides  basic  core  principles  in  science,  process  (design),  and  logical  thinking 

•  yes,  less  difficult  to  “cross  train”  into  various  technical  management  and  financial 
disciplines  than  vice-versa 
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•  yes,  most  of  what  we  manage  in  government  is  technical  and  without  capacity  to 
understand  “tech,”  the  manager  is  not  very  effective  and  is  vulnerable 

•  yes,  a  technical  degree  gives  a  person  the  background  to  understand  the  technical 
nature  of  the  work  and  visualize  its  impact  on  real  world  operations;  someone  without 
a  technical  degree  would  have  more  difficulty  understanding  the  technical  value  and 
making  decisions  to  continue  or  stop  a  given  technology  program 

•  yes,  having  a  technical  degree,  as  a  minimum,  gives  credibility  with  a  contractor 

•  yes,  technical  management  relies  on  sound  technical  decisions;  a  technical  manager 
needs  the  tech  background  to  understand,  lead,  and  represent  the  area 

•  yes,  managing  S&T  requires  basic  knowledge  of  the  management  area  (in  this  case 
it’s  managing  S&T) 

•  yes,  important  to  have  technical  background  when  making  decisions  such  as  funding 
decisions,  manpower  allocations,  etc. 

•  yes,  absolutely:  those  with  said  degrees  won’t  respect  or  trust  you;  you  need  to  be 
able  to  smell  the  BS 
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